[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:34 dist-test-slave-j38h krb5kdc[2087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:34 dist-test-slave-j38h krb5kdc[2087](info): set up 2 sockets
Sep 24 10:39:34 dist-test-slave-j38h krb5kdc[2087](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 10:39:35.895162 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:39:35 dist-test-slave-j38h krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:39:36.015903 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:42701
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:42701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:36.124351 2100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:36.124670 2100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:36.124759 2100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:36.128755 2100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:36.128863 2100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:36.128902 2100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:36.128940 2100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:36.128988 2100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:36.134402 2100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:42701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:42701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:36.135565 2100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:36.136655 2100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:39:36.145694 2100 server_base.cc:1034] running on GCE node
W20260924 10:39:36.147665 2105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:36.147606 2108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:36.147792 2106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:36.148504 2100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:36.149717 2100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:36.150916 2100 hybrid_clock.cc:648] HybridClock initialized: now 1790246376150879 us; error 51 us; skew 500 ppm
Sep 24 10:39:36 dist-test-slave-j38h krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246376, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:36.153786 2100 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:39:36.154662 2100 webserver.cc:460] Webserver started at http://127.2.8.62:42483/ using document root <none> and password file <none>
I20260924 10:39:36.155230 2100 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:36.155378 2100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:36.155611 2100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:36.157706 2100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "e3fecfc0d1e94be09e41e826e9d28ed8"
format_stamp: "Formatted at 2026-09-24 10:39:36 on dist-test-slave-j38h"
server_key: "c40d5b4652bd8611e9fb5cc3ce341a98"
server_key_iv: "f98e2db13e10f2f7ee2172e70ddce7f7"
server_key_version: "encryptionkey@0"
I20260924 10:39:36.158298 2100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "e3fecfc0d1e94be09e41e826e9d28ed8"
format_stamp: "Formatted at 2026-09-24 10:39:36 on dist-test-slave-j38h"
server_key: "c40d5b4652bd8611e9fb5cc3ce341a98"
server_key_iv: "f98e2db13e10f2f7ee2172e70ddce7f7"
server_key_version: "encryptionkey@0"
I20260924 10:39:36.162045 2100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:39:36.164484 2115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:36.165755 2100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:39:36.165892 2100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "e3fecfc0d1e94be09e41e826e9d28ed8"
format_stamp: "Formatted at 2026-09-24 10:39:36 on dist-test-slave-j38h"
server_key: "c40d5b4652bd8611e9fb5cc3ce341a98"
server_key_iv: "f98e2db13e10f2f7ee2172e70ddce7f7"
server_key_version: "encryptionkey@0"
I20260924 10:39:36.166005 2100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:36.183730 2100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:36.186648 2100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:36.186874 2100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:36.194039 2100 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:42701
I20260924 10:39:36.194064 2167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:42701 every 8 connection(s)
I20260924 10:39:36.194949 2100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:36.197670 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:36.202195 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 2100
I20260924 10:39:36.202348 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:39:36.203058 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8: Bootstrap starting.
I20260924 10:39:36.203670 2080 external_mini_cluster.cc:1428] Setting key ee27716c7897ac3bc3d176e9e41e30b2
I20260924 10:39:36.205489 2168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:36.206441 2168 log.cc:824] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:36.208640 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8: No bootstrap required, opened a new log
I20260924 10:39:36.212575 2168 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fecfc0d1e94be09e41e826e9d28ed8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42701 } }
I20260924 10:39:36.212841 2168 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:36.212898 2168 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fecfc0d1e94be09e41e826e9d28ed8, State: Initialized, Role: FOLLOWER
I20260924 10:39:36.213380 2168 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fecfc0d1e94be09e41e826e9d28ed8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42701 } }
I20260924 10:39:36.213503 2168 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:36.213593 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:36.213711 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:36.214555 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fecfc0d1e94be09e41e826e9d28ed8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42701 } }
I20260924 10:39:36.214881 2168 leader_election.cc:302] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3fecfc0d1e94be09e41e826e9d28ed8; no voters:
I20260924 10:39:36.215159 2168 leader_election.cc:288] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:36.215276 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:36.215499 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [term 1 LEADER]: Becoming Leader. State: Replica: e3fecfc0d1e94be09e41e826e9d28ed8, State: Running, Role: LEADER
I20260924 10:39:36.215801 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fecfc0d1e94be09e41e826e9d28ed8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42701 } }
I20260924 10:39:36.216427 2168 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:36.217516 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3fecfc0d1e94be09e41e826e9d28ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fecfc0d1e94be09e41e826e9d28ed8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42701 } } }
I20260924 10:39:36.217682 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:36.217517 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3fecfc0d1e94be09e41e826e9d28ed8. Latest consensus state: current_term: 1 leader_uuid: "e3fecfc0d1e94be09e41e826e9d28ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fecfc0d1e94be09e41e826e9d28ed8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42701 } } }
I20260924 10:39:36.217907 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:36.218221 2182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:36.221443 2182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 10:39:36 dist-test-slave-j38h krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:36.227250 2182 catalog_manager.cc:1347] Generated new cluster ID: f3da1a212f0f4860b5498ea7c74bd1a2
I20260924 10:39:36.227306 2182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:36.231556 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:36.206778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.2.8.62:42701)
0924 10:39:36.208382 (+ 1604us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:36.208402 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:36.208443 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:39:36.212116 (+ 3673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:36.212137 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:36.212514 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:36.212877 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:36.215418 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:36.217075 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:36.219925 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:36.220583 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:36.220755 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:36.225710 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:36.225718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:36.225741 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:36.225774 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:36.228282 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:36.228805 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:36.228817 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:36.228821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:36.228924 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:36.229308 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:36.229311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:36.229313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:36.229702 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:36.229930 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:36.230461 (+ 531us) server_negotiation.cc:299] Negotiation successful
0924 10:39:36.230720 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":140,"threads_started":1}
I20260924 10:39:36.235201 2182 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:36.236057 2182 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:36.246124 2182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3fecfc0d1e94be09e41e826e9d28ed8: Generated new TSK 0
I20260924 10:39:36.246814 2182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:39:36.280052 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:42701
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:36.398665 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:36.398929 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:36.398979 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:36.403079 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:36.403136 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:36.403195 2194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:39:36.408495 2194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:42701
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:36.409613 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:36.410640 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:36.420194 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:36.421890 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:36.422472 2194 server_base.cc:1034] running on GCE node
W20260924 10:39:36.422545 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:36.422938 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:36.423621 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:36.424778 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790246376424758 us; error 32 us; skew 500 ppm
Sep 24 10:39:36 dist-test-slave-j38h krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246376, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:36.427747 2194 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:39:36.428635 2194 webserver.cc:460] Webserver started at http://127.2.8.1:46267/ using document root <none> and password file <none>
I20260924 10:39:36.429255 2194 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:36.429386 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:36.429608 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:36.431576 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "9c501930613449dd909697568fcef039"
format_stamp: "Formatted at 2026-09-24 10:39:36 on dist-test-slave-j38h"
server_key: "0a5714f6ce74ef53dc61e83d86424d31"
server_key_iv: "cb4ad16e08e126dac4892dadf71dd4a3"
server_key_version: "encryptionkey@0"
I20260924 10:39:36.432118 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "9c501930613449dd909697568fcef039"
format_stamp: "Formatted at 2026-09-24 10:39:36 on dist-test-slave-j38h"
server_key: "0a5714f6ce74ef53dc61e83d86424d31"
server_key_iv: "cb4ad16e08e126dac4892dadf71dd4a3"
server_key_version: "encryptionkey@0"
I20260924 10:39:36.435731 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:39:36.438148 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:36.439476 2194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:39:36.439595 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "9c501930613449dd909697568fcef039"
format_stamp: "Formatted at 2026-09-24 10:39:36 on dist-test-slave-j38h"
server_key: "0a5714f6ce74ef53dc61e83d86424d31"
server_key_iv: "cb4ad16e08e126dac4892dadf71dd4a3"
server_key_version: "encryptionkey@0"
I20260924 10:39:36.439700 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:36.462849 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:36.465907 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:36.466058 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:36.466580 2194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:36.467355 2194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:36.467399 2194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:36.467438 2194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:36.467459 2194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:36.476002 2194 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:38901
I20260924 10:39:36.476053 2322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:38901 every 8 connection(s)
I20260924 10:39:36.476944 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:39:36.486819 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 2194
I20260924 10:39:36.486948 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
Sep 24 10:39:36 dist-test-slave-j38h krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246376, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:36.487219 2080 external_mini_cluster.cc:1428] Setting key 207d3edce45ec579f64bc217ac68671b
I20260924 10:39:36.491099 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:36.478980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:42485 (local address 127.2.8.62:42701)
0924 10:39:36.479166 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:36.479171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:36.480213 (+ 1042us) server_negotiation.cc:407] Connection header received
0924 10:39:36.481796 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:36.481799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:36.481865 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:36.481955 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:36.483103 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:36.483648 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:36.484776 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:36.485084 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:36.485195 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:36.487577 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:36.487580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:36.487583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:36.487615 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:36.489156 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:36.489631 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:36.489636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:36.489638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:36.489720 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:36.490057 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:36.490060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:36.490062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:36.490230 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:36.490332 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:36.490804 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 10:39:36.490933 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 10:39:36.491876 2325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:36.479475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:42701 (local address 127.2.8.1:42485)
0924 10:39:36.480044 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:36.480101 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:36.481558 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:36.482133 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:36.482140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:36.482488 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:36.482958 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:36.482971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:36.483769 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:36.483772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:36.484650 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:36.484659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:36.485180 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:36.485234 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:36.485409 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:36.485952 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:36.485967 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:36.487422 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:36.489267 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:36.489273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:36.489287 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:36.489531 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:36.489810 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:36.489814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:36.489816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:36.489954 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:36.490331 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:36.490337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:36.490569 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 10:39:36.490751 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 10:39:36.491072 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":140,"threads_started":1}
I20260924 10:39:36.493098 2323 heartbeater.cc:347] Connected to a master server at 127.2.8.62:42701
I20260924 10:39:36.493333 2323 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:36.493806 2323 heartbeater.cc:511] Master 127.2.8.62:42701 requested a full tablet report, sending...
I20260924 10:39:36.495270 2132 ts_manager.cc:194] Registered new tserver with Master: 9c501930613449dd909697568fcef039 (127.2.8.1:38901)
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
I20260924 10:39:36.496435 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:42485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:39:36.516263 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:42701
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:36.624379 2328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:36.624624 2328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:36.624694 2328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:36.628708 2328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:36.628762 2328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:36.628877 2328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:39:36.634114 2328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:42701
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:36.635278 2328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:36.636298 2328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:36.645604 2334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:36.645654 2333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:36.645622 2336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:36.645857 2328 server_base.cc:1034] running on GCE node
I20260924 10:39:36.646270 2328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:36.646940 2328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:36.648094 2328 hybrid_clock.cc:648] HybridClock initialized: now 1790246376648076 us; error 29 us; skew 500 ppm
Sep 24 10:39:36 dist-test-slave-j38h krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246376, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:36.650902 2328 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:39:36.651821 2328 webserver.cc:460] Webserver started at http://127.2.8.2:38863/ using document root <none> and password file <none>
I20260924 10:39:36.652374 2328 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:36.652462 2328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:36.652685 2328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:36.654848 2328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "0b5a1e5f3c334ead9d5176624d26641c"
format_stamp: "Formatted at 2026-09-24 10:39:36 on dist-test-slave-j38h"
server_key: "a288c7bbc23257cd272022331d25f0a3"
server_key_iv: "02f0c330bc012b4d37a04ac1013d6cb1"
server_key_version: "encryptionkey@0"
I20260924 10:39:36.655414 2328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "0b5a1e5f3c334ead9d5176624d26641c"
format_stamp: "Formatted at 2026-09-24 10:39:36 on dist-test-slave-j38h"
server_key: "a288c7bbc23257cd272022331d25f0a3"
server_key_iv: "02f0c330bc012b4d37a04ac1013d6cb1"
server_key_version: "encryptionkey@0"
I20260924 10:39:36.659111 2328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:39:36.661541 2343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:36.662804 2328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:39:36.662948 2328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "0b5a1e5f3c334ead9d5176624d26641c"
format_stamp: "Formatted at 2026-09-24 10:39:36 on dist-test-slave-j38h"
server_key: "a288c7bbc23257cd272022331d25f0a3"
server_key_iv: "02f0c330bc012b4d37a04ac1013d6cb1"
server_key_version: "encryptionkey@0"
I20260924 10:39:36.663056 2328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:36.676158 2328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:36.679175 2328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:36.679378 2328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:36.679987 2328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:36.680820 2328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:36.680896 2328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:36.681013 2328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:36.681069 2328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:36.689980 2328 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:41383
I20260924 10:39:36.690004 2456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:41383 every 8 connection(s)
I20260924 10:39:36.690861 2328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:39:36.692638 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 2328
I20260924 10:39:36.692763 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:39:36.693092 2080 external_mini_cluster.cc:1428] Setting key 88a2ed91e8187de70d0a0819370fda89
Sep 24 10:39:36 dist-test-slave-j38h krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246376, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
I20260924 10:39:36.706218 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:36.692697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:52545 (local address 127.2.8.62:42701)
0924 10:39:36.692879 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:36.692883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:36.693818 (+ 935us) server_negotiation.cc:407] Connection header received
0924 10:39:36.695202 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:36.695209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:36.695267 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:36.695357 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:36.696804 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:36.697421 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:36.699222 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:36.699440 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:36.699520 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:36.702420 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:36.702423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:36.702425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:36.702448 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:36.703942 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:36.704481 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:36.704486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:36.704488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:36.704571 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:36.705009 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:36.705015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:36.705017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:36.705204 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:36.705307 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:36.705902 (+ 595us) server_negotiation.cc:299] Negotiation successful
0924 10:39:36.706029 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 10:39:36.707005 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:36.693209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:42701 (local address 127.2.8.2:52545)
0924 10:39:36.693675 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:36.693727 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:36.694998 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:36.695484 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:36.695492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:36.695914 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:36.696647 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:36.696670 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:36.697546 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:36.697552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:36.699092 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:36.699106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:36.699543 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:36.699604 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:36.699845 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:36.700530 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:36.700553 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:36.702240 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:36.704073 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:36.704079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:36.704094 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:36.704350 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:36.704665 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:36.704669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:36.704671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:36.704840 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:36.705315 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:36.705323 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:36.705620 (+ 297us) client_negotiation.cc:769] Sending connection context
0924 10:39:36.705844 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 10:39:36.706167 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":127,"threads_started":1}
I20260924 10:39:36.708324 2457 heartbeater.cc:347] Connected to a master server at 127.2.8.62:42701
I20260924 10:39:36.708581 2457 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:36.709133 2457 heartbeater.cc:511] Master 127.2.8.62:42701 requested a full tablet report, sending...
I20260924 10:39:36.710206 2132 ts_manager.cc:194] Registered new tserver with Master: 0b5a1e5f3c334ead9d5176624d26641c (127.2.8.2:41383)
I20260924 10:39:36.710790 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:52545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:39:36.724228 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:42701
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:36.835850 2462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:36.836140 2462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:36.836185 2462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:36.840397 2462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:36.840463 2462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:36.840526 2462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:39:36.849411 2462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:42701
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:36.850617 2462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:36.851712 2462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:36.863634 2470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:36.863624 2468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:36.863648 2467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:36.863981 2462 server_base.cc:1034] running on GCE node
I20260924 10:39:36.864357 2462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:36.864914 2462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:36.866148 2462 hybrid_clock.cc:648] HybridClock initialized: now 1790246376866133 us; error 26 us; skew 500 ppm
Sep 24 10:39:36 dist-test-slave-j38h krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246376, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:36.868923 2462 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:39:36.869958 2462 webserver.cc:460] Webserver started at http://127.2.8.3:41727/ using document root <none> and password file <none>
I20260924 10:39:36.870566 2462 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:36.870627 2462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:36.870848 2462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:36.872850 2462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "0ac1648a399e404c956a42a6bc250c6e"
format_stamp: "Formatted at 2026-09-24 10:39:36 on dist-test-slave-j38h"
server_key: "d75cf61dbc367a274effd8eeff1f75c9"
server_key_iv: "db191460542419d9abe336e6b3c8a8c5"
server_key_version: "encryptionkey@0"
I20260924 10:39:36.873492 2462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "0ac1648a399e404c956a42a6bc250c6e"
format_stamp: "Formatted at 2026-09-24 10:39:36 on dist-test-slave-j38h"
server_key: "d75cf61dbc367a274effd8eeff1f75c9"
server_key_iv: "db191460542419d9abe336e6b3c8a8c5"
server_key_version: "encryptionkey@0"
I20260924 10:39:36.877096 2462 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:39:36.879500 2477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:36.880807 2462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:36.880946 2462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "0ac1648a399e404c956a42a6bc250c6e"
format_stamp: "Formatted at 2026-09-24 10:39:36 on dist-test-slave-j38h"
server_key: "d75cf61dbc367a274effd8eeff1f75c9"
server_key_iv: "db191460542419d9abe336e6b3c8a8c5"
server_key_version: "encryptionkey@0"
I20260924 10:39:36.881110 2462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:36.931228 2462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:36.934543 2462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:36.934756 2462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:36.935396 2462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:36.936228 2462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:36.936298 2462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:36.936365 2462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:36.936416 2462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:36.946350 2462 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:38777
I20260924 10:39:36.946367 2590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:38777 every 8 connection(s)
I20260924 10:39:36.947317 2462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:39:36.951205 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 2462
I20260924 10:39:36.951308 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:39:36.951555 2080 external_mini_cluster.cc:1428] Setting key fd76dc37961c500d64d5f2c4d5355fe3
Sep 24 10:39:36 dist-test-slave-j38h krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246376, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:36.961710 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:36.949144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:36137 (local address 127.2.8.62:42701)
0924 10:39:36.949279 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:36.949282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:36.950119 (+ 837us) server_negotiation.cc:407] Connection header received
0924 10:39:36.951784 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:36.951787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:36.951844 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:36.951909 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:36.953253 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:36.954020 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:36.955273 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:36.955510 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:36.955620 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:36.958212 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:36.958215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:36.958218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:36.958246 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:36.959586 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:36.960108 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:36.960111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:36.960113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:36.960198 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:36.960539 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:36.960542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:36.960544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:36.960725 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:36.960846 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:36.961425 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 10:39:36.961541 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:39:36.962512 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:36.949546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:42701 (local address 127.2.8.3:36137)
0924 10:39:36.949975 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:36.950018 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:36.951523 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:36.952130 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:36.952138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:36.952495 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:36.953114 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:36.953130 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:36.954143 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:36.954146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:36.955138 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:36.955147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:36.955607 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:36.955659 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:36.955832 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:36.956470 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:36.956486 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:36.958046 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:36.959703 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:36.959709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:36.959725 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:36.960000 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:36.960291 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:36.960294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:36.960303 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:36.960437 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:36.960846 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:36.960853 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:36.961163 (+ 310us) client_negotiation.cc:769] Sending connection context
0924 10:39:36.961356 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 10:39:36.961676 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
I20260924 10:39:36.963635 2591 heartbeater.cc:347] Connected to a master server at 127.2.8.62:42701
I20260924 10:39:36.963901 2591 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:36.964432 2591 heartbeater.cc:511] Master 127.2.8.62:42701 requested a full tablet report, sending...
I20260924 10:39:36.965619 2132 ts_manager.cc:194] Registered new tserver with Master: 0ac1648a399e404c956a42a6bc250c6e (127.2.8.3:38777)
I20260924 10:39:36.966342 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:36137
I20260924 10:39:36.966492 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:39:36 dist-test-slave-j38h krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246376, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:39:36 dist-test-slave-j38h krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246376, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.8.1@KRBTEST.COM
I20260924 10:39:36.992358 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:36.982635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.2.8.1:38901)
0924 10:39:36.982912 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:36.982925 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:36.982949 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:39:36.983032 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:36.983037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:36.983226 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:36.983387 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:36.984035 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:36.984910 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:36.985914 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:36.986276 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:36.986410 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:36.988769 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:36.988777 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:36.988790 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:36.988813 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:36.990631 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:36.991036 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:36.991042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:36.991043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:36.991127 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:36.991421 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:36.991423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:36.991424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:36.991652 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:36.991801 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:36.991980 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 10:39:36.992139 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
W20260924 10:39:36.993407 2237 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55858
Sep 24 10:39:36 dist-test-slave-j38h krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:39:37 dist-test-slave-j38h krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.1@KRBTEST.COM
I20260924 10:39:37.019372 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:37.010796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.2.8.1:38901)
0924 10:39:37.010963 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:37.010967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:37.010986 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:39:37.011126 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:37.011129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:37.011170 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:37.011240 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:37.011929 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:37.012464 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:37.013387 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:37.013638 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:37.013750 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:37.016047 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:37.016050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:37.016052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:37.016075 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:37.017835 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:37.018186 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:37.018188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:37.018189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:37.018259 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:37.018518 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:37.018520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:37.018521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:37.018722 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:37.018820 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:37.019076 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 10:39:37.019167 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:39:37.020598 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 2194
I20260924 10:39:37.026288 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 2328
I20260924 10:39:37.032271 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 2462
I20260924 10:39:37.037191 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 2100
2026-09-24T10:39:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2211 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:37 dist-test-slave-j38h krb5kdc[2605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:37 dist-test-slave-j38h krb5kdc[2605](info): set up 2 sockets
Sep 24 10:39:37 dist-test-slave-j38h krb5kdc[2605](info): commencing operation
krb5kdc: starting...
W20260924 10:39:38.068576 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:39:38 dist-test-slave-j38h krb5kdc[2605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:39:38.189996 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:43957
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:43957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:38.299614 2618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:38.299942 2618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:38.300047 2618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:38.304073 2618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:38.304183 2618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:38.304229 2618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:38.304291 2618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:38.304330 2618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:38.309643 2618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:43957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:43957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:38.310828 2618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:38.311866 2618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:38.320457 2624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:38.320499 2623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:38.320659 2618 server_base.cc:1034] running on GCE node
W20260924 10:39:38.320457 2626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:38.321396 2618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:38.322543 2618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:38.323761 2618 hybrid_clock.cc:648] HybridClock initialized: now 1790246378323748 us; error 26 us; skew 500 ppm
Sep 24 10:39:38 dist-test-slave-j38h krb5kdc[2605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246378, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:38.326615 2618 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:39:38.327553 2618 webserver.cc:460] Webserver started at http://127.2.8.62:45465/ using document root <none> and password file <none>
I20260924 10:39:38.328118 2618 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:38.328208 2618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:38.328437 2618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:38.330588 2618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "ec03eb6f0c6d475ca01e8825897ee0c5"
format_stamp: "Formatted at 2026-09-24 10:39:38 on dist-test-slave-j38h"
server_key: "8f9bd869ba1c466647f77abb152a82c3"
server_key_iv: "6c1740e1655dbd2db999f01364b5b421"
server_key_version: "encryptionkey@0"
I20260924 10:39:38.331223 2618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "ec03eb6f0c6d475ca01e8825897ee0c5"
format_stamp: "Formatted at 2026-09-24 10:39:38 on dist-test-slave-j38h"
server_key: "8f9bd869ba1c466647f77abb152a82c3"
server_key_iv: "6c1740e1655dbd2db999f01364b5b421"
server_key_version: "encryptionkey@0"
I20260924 10:39:38.335098 2618 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 10:39:38.337685 2633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:38.339016 2618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:39:38.339174 2618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "ec03eb6f0c6d475ca01e8825897ee0c5"
format_stamp: "Formatted at 2026-09-24 10:39:38 on dist-test-slave-j38h"
server_key: "8f9bd869ba1c466647f77abb152a82c3"
server_key_iv: "6c1740e1655dbd2db999f01364b5b421"
server_key_version: "encryptionkey@0"
I20260924 10:39:38.339296 2618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:38.351503 2618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:38.354831 2618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:38.355042 2618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:38.362149 2618 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:43957
I20260924 10:39:38.362170 2685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:43957 every 8 connection(s)
I20260924 10:39:38.363121 2618 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:38.365813 2686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:38.366292 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 2618
I20260924 10:39:38.366410 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:39:38.366695 2080 external_mini_cluster.cc:1428] Setting key a5b1f24390366c4c6ddd50913f00a8e9
I20260924 10:39:38.372310 2686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5: Bootstrap starting.
I20260924 10:39:38.374538 2686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:38.375402 2686 log.cc:824] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:39:38 dist-test-slave-j38h krb5kdc[2605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:38.377740 2686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5: No bootstrap required, opened a new log
I20260924 10:39:38.381320 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:38.368193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60860 (local address 127.2.8.62:43957)
0924 10:39:38.368615 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:38.368628 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:38.368661 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:39:38.369375 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:38.369392 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:38.369708 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:38.370046 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:38.370802 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:38.371781 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:38.372681 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:38.373212 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:38.373370 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:38.376111 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:38.376117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:38.376152 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:38.376178 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:38.378155 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:38.378571 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:38.378576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:38.378582 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:38.378718 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:38.378994 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:38.378997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:38.378999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:38.379499 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:38.379683 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:38.379901 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 10:39:38.380288 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
I20260924 10:39:38.382400 2686 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec03eb6f0c6d475ca01e8825897ee0c5" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43957 } }
I20260924 10:39:38.382664 2686 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:38.382745 2686 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec03eb6f0c6d475ca01e8825897ee0c5, State: Initialized, Role: FOLLOWER
I20260924 10:39:38.383216 2686 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec03eb6f0c6d475ca01e8825897ee0c5" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43957 } }
I20260924 10:39:38.383338 2686 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:38.383424 2686 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:38.383559 2686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:38.384500 2686 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec03eb6f0c6d475ca01e8825897ee0c5" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43957 } }
I20260924 10:39:38.384843 2686 leader_election.cc:302] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec03eb6f0c6d475ca01e8825897ee0c5; no voters:
I20260924 10:39:38.385205 2686 leader_election.cc:288] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:38.385316 2691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:38.385550 2691 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [term 1 LEADER]: Becoming Leader. State: Replica: ec03eb6f0c6d475ca01e8825897ee0c5, State: Running, Role: LEADER
I20260924 10:39:38.385888 2691 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec03eb6f0c6d475ca01e8825897ee0c5" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43957 } }
I20260924 10:39:38.386430 2686 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:38.387528 2692 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec03eb6f0c6d475ca01e8825897ee0c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec03eb6f0c6d475ca01e8825897ee0c5" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43957 } } }
I20260924 10:39:38.387670 2692 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:38.387961 2693 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec03eb6f0c6d475ca01e8825897ee0c5. Latest consensus state: current_term: 1 leader_uuid: "ec03eb6f0c6d475ca01e8825897ee0c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec03eb6f0c6d475ca01e8825897ee0c5" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43957 } } }
I20260924 10:39:38.388062 2700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:38.388096 2693 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:38.390838 2700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:38.395975 2700 catalog_manager.cc:1347] Generated new cluster ID: 06a0d627a58f4b93acfa02ac6f018744
I20260924 10:39:38.396044 2700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:38.410048 2700 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:38.411310 2700 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:38.422930 2700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec03eb6f0c6d475ca01e8825897ee0c5: Generated new TSK 0
I20260924 10:39:38.423641 2700 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:39:38.457036 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43957
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:38.566419 2712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:38.566710 2712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:38.566787 2712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:38.570834 2712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:38.570920 2712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:38.571012 2712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:39:38.576294 2712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:38.577443 2712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:38.578523 2712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:38.588194 2717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:38.588194 2718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:38.588325 2720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:38.588579 2712 server_base.cc:1034] running on GCE node
I20260924 10:39:38.588960 2712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:38.589627 2712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:38.590804 2712 hybrid_clock.cc:648] HybridClock initialized: now 1790246378590781 us; error 32 us; skew 500 ppm
Sep 24 10:39:38 dist-test-slave-j38h krb5kdc[2605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246378, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:38.593833 2712 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:39:38.594930 2712 webserver.cc:460] Webserver started at http://127.2.8.1:40081/ using document root <none> and password file <none>
I20260924 10:39:38.595548 2712 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:38.595649 2712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:38.595903 2712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:38.598142 2712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "e95a887b02e34681bc7cb30b03bd18e5"
format_stamp: "Formatted at 2026-09-24 10:39:38 on dist-test-slave-j38h"
server_key: "af81f284a6238416c984ea66e6eb07c5"
server_key_iv: "e86fa7443b56a787b89c83c5ed49d44c"
server_key_version: "encryptionkey@0"
I20260924 10:39:38.598740 2712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "e95a887b02e34681bc7cb30b03bd18e5"
format_stamp: "Formatted at 2026-09-24 10:39:38 on dist-test-slave-j38h"
server_key: "af81f284a6238416c984ea66e6eb07c5"
server_key_iv: "e86fa7443b56a787b89c83c5ed49d44c"
server_key_version: "encryptionkey@0"
I20260924 10:39:38.602708 2712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 10:39:38.605463 2727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:38.606725 2712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:39:38.606838 2712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "e95a887b02e34681bc7cb30b03bd18e5"
format_stamp: "Formatted at 2026-09-24 10:39:38 on dist-test-slave-j38h"
server_key: "af81f284a6238416c984ea66e6eb07c5"
server_key_iv: "e86fa7443b56a787b89c83c5ed49d44c"
server_key_version: "encryptionkey@0"
I20260924 10:39:38.606945 2712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:38.616654 2712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:38.619598 2712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:38.619750 2712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:38.620261 2712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:38.621148 2712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:38.621194 2712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:38.621233 2712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:38.621249 2712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:38.630508 2712 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:33411
I20260924 10:39:38.630546 2840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:33411 every 8 connection(s)
I20260924 10:39:38.631400 2712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:39:38.633160 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 2712
I20260924 10:39:38.633289 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:39:38.633602 2080 external_mini_cluster.cc:1428] Setting key 85abd8ae8c09ae3ce3aec04cccc12def
Sep 24 10:39:38 dist-test-slave-j38h krb5kdc[2605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246378, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
I20260924 10:39:38.647217 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:38.633404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:34697 (local address 127.2.8.62:43957)
0924 10:39:38.633578 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:38.633583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:38.634415 (+ 832us) server_negotiation.cc:407] Connection header received
0924 10:39:38.635923 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:38.635927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:38.635981 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:38.636065 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:38.637560 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:38.638076 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:38.639877 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:38.640111 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:38.640186 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:38.643124 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:38.643126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:38.643128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:38.643150 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:38.644754 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:38.645505 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:38.645512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:38.645517 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:38.645643 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:38.646068 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:38.646071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:38.646073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:38.646272 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:38.646385 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:38.646905 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 10:39:38.647032 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 10:39:38.648051 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:38.633779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43957 (local address 127.2.8.1:34697)
0924 10:39:38.634270 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:38.634320 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:38.635719 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:38.636244 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:38.636251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:38.636678 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:38.637402 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:38.637426 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:38.638202 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:38.638208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:38.639739 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:38.639753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:38.640208 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:38.640269 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:38.640516 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:38.641235 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:38.641251 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:38.642946 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:38.644883 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:38.644889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:38.644904 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:38.645332 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:38.645753 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:38.645757 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:38.645759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:38.645929 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:38.646382 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:38.646388 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:38.646642 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 10:39:38.646846 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 10:39:38.647178 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":106,"threads_started":1}
I20260924 10:39:38.649392 2841 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43957
I20260924 10:39:38.649649 2841 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:38.650274 2841 heartbeater.cc:511] Master 127.2.8.62:43957 requested a full tablet report, sending...
I20260924 10:39:38.652072 2650 ts_manager.cc:194] Registered new tserver with Master: e95a887b02e34681bc7cb30b03bd18e5 (127.2.8.1:33411)
I20260924 10:39:38.653707 2650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:34697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:39:38.661845 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43957
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:38.773379 2846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:38.773644 2846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:38.773689 2846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:38.777709 2846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:38.777765 2846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:38.777825 2846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:39:38.783006 2846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:38.783991 2846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:38.785244 2846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:39:38.795059 2846 server_base.cc:1034] running on GCE node
W20260924 10:39:38.795104 2854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:38.795087 2851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:38.795042 2852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:38.795709 2846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:38.796278 2846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:38.797456 2846 hybrid_clock.cc:648] HybridClock initialized: now 1790246378797423 us; error 44 us; skew 500 ppm
Sep 24 10:39:38 dist-test-slave-j38h krb5kdc[2605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246378, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:38.800235 2846 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:39:38.801222 2846 webserver.cc:460] Webserver started at http://127.2.8.2:45945/ using document root <none> and password file <none>
I20260924 10:39:38.801770 2846 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:38.801829 2846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:38.802033 2846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:38.803972 2846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "6793fd8ebedf4af8a027bdcd616f1d1d"
format_stamp: "Formatted at 2026-09-24 10:39:38 on dist-test-slave-j38h"
server_key: "fef79b8fab2e3f060ac7fd1be3341714"
server_key_iv: "289a63cb97324d7f1e464cf529a893d0"
server_key_version: "encryptionkey@0"
I20260924 10:39:38.804517 2846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "6793fd8ebedf4af8a027bdcd616f1d1d"
format_stamp: "Formatted at 2026-09-24 10:39:38 on dist-test-slave-j38h"
server_key: "fef79b8fab2e3f060ac7fd1be3341714"
server_key_iv: "289a63cb97324d7f1e464cf529a893d0"
server_key_version: "encryptionkey@0"
I20260924 10:39:38.808151 2846 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 10:39:38.810652 2861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:38.812026 2846 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:39:38.812151 2846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "6793fd8ebedf4af8a027bdcd616f1d1d"
format_stamp: "Formatted at 2026-09-24 10:39:38 on dist-test-slave-j38h"
server_key: "fef79b8fab2e3f060ac7fd1be3341714"
server_key_iv: "289a63cb97324d7f1e464cf529a893d0"
server_key_version: "encryptionkey@0"
I20260924 10:39:38.812305 2846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:38.830904 2846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:38.834208 2846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:38.834537 2846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:38.835191 2846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:38.836081 2846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:38.836131 2846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:38.836174 2846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:38.836238 2846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:38.846027 2846 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:33607
I20260924 10:39:38.846045 2974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:33607 every 8 connection(s)
I20260924 10:39:38.846985 2846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:39:38.848256 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 2846
I20260924 10:39:38.848351 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:39:38.848641 2080 external_mini_cluster.cc:1428] Setting key d4ddb1a58104152c20edd731c91e3d3e
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
Sep 24 10:39:38 dist-test-slave-j38h krb5kdc[2605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246378, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:38.867341 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:38.849049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:58485 (local address 127.2.8.62:43957)
0924 10:39:38.849243 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:38.849249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:38.850133 (+ 884us) server_negotiation.cc:407] Connection header received
0924 10:39:38.851706 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:38.851710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:38.851801 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:38.851940 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:38.854508 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:38.855637 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:38.857556 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:38.857954 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:38.858073 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:38.863190 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:38.863193 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:38.863195 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:38.863221 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:38.864868 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:38.865489 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:38.865495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:38.865500 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:38.865594 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:38.866004 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:38.866010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:38.866015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:38.866276 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:38.866391 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:38.866933 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 10:39:38.867080 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:39:38.868139 2977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:38.849511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43957 (local address 127.2.8.2:58485)
0924 10:39:38.849978 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:38.850031 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:38.851439 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:38.852403 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:38.852415 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:38.853225 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:38.854218 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:38.854268 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:38.855819 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:38.855824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:38.857417 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:38.857431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:38.858071 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:38.858131 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:38.858362 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:38.859039 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:38.859066 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:38.862971 (+ 3905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:38.865046 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:38.865052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:38.865067 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:38.865363 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:38.865687 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:38.865689 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:38.865690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:38.865870 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:38.866384 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:38.866389 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:38.866646 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 10:39:38.866821 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 10:39:38.867136 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":135,"threads_started":1}
I20260924 10:39:38.869524 2975 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43957
I20260924 10:39:38.869791 2975 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:38.870379 2975 heartbeater.cc:511] Master 127.2.8.62:43957 requested a full tablet report, sending...
I20260924 10:39:38.871532 2650 ts_manager.cc:194] Registered new tserver with Master: 6793fd8ebedf4af8a027bdcd616f1d1d (127.2.8.2:33607)
I20260924 10:39:38.872293 2650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:58485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:39:38.878320 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43957
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:38.987825 2980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:38.988170 2980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:38.988263 2980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:38.992585 2980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:38.992679 2980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:38.992774 2980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:39:38.998121 2980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43957
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:38.999205 2980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:39.000309 2980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:39.010082 2986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:39.010134 2985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:39.010134 2988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:39.010318 2980 server_base.cc:1034] running on GCE node
I20260924 10:39:39.010819 2980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:39.011518 2980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:39.012702 2980 hybrid_clock.cc:648] HybridClock initialized: now 1790246379012687 us; error 27 us; skew 500 ppm
Sep 24 10:39:39 dist-test-slave-j38h krb5kdc[2605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246379, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:39.015658 2980 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:39:39.016635 2980 webserver.cc:460] Webserver started at http://127.2.8.3:42089/ using document root <none> and password file <none>
I20260924 10:39:39.017279 2980 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:39.017369 2980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:39.017607 2980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:39.019588 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "b0ef899ffff147fea9a0ddbb04b025e5"
format_stamp: "Formatted at 2026-09-24 10:39:39 on dist-test-slave-j38h"
server_key: "ba0a6d5a0f0c2c67930d616783f29c1b"
server_key_iv: "158074e472fe97ebffba9a1208f6402f"
server_key_version: "encryptionkey@0"
I20260924 10:39:39.020169 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "b0ef899ffff147fea9a0ddbb04b025e5"
format_stamp: "Formatted at 2026-09-24 10:39:39 on dist-test-slave-j38h"
server_key: "ba0a6d5a0f0c2c67930d616783f29c1b"
server_key_iv: "158074e472fe97ebffba9a1208f6402f"
server_key_version: "encryptionkey@0"
I20260924 10:39:39.023896 2980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 10:39:39.026331 2995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:39.027470 2980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:39:39.027598 2980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "b0ef899ffff147fea9a0ddbb04b025e5"
format_stamp: "Formatted at 2026-09-24 10:39:39 on dist-test-slave-j38h"
server_key: "ba0a6d5a0f0c2c67930d616783f29c1b"
server_key_iv: "158074e472fe97ebffba9a1208f6402f"
server_key_version: "encryptionkey@0"
I20260924 10:39:39.027704 2980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:39.043951 2980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:39.047493 2980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:39.047698 2980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:39.048310 2980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:39.049266 2980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:39.049328 2980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:39.049404 2980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:39.049446 2980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:39.058892 2980 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:33951
I20260924 10:39:39.058916 3108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:33951 every 8 connection(s)
I20260924 10:39:39.059834 2980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:39:39.064517 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 2980
I20260924 10:39:39.064693 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:39:39.065028 2080 external_mini_cluster.cc:1428] Setting key 902047702526064db9274b4da9d8b631
Sep 24 10:39:39 dist-test-slave-j38h krb5kdc[2605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246379, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:39.074782 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.061911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:56207 (local address 127.2.8.62:43957)
0924 10:39:39.062059 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:39.062063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:39.062895 (+ 832us) server_negotiation.cc:407] Connection header received
0924 10:39:39.064455 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:39.064458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:39.064507 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:39.064602 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:39.066429 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.067169 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.068488 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.068698 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.068779 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.071424 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:39.071426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:39.071428 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:39.071453 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:39.072901 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:39.073440 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:39.073443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:39.073444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:39.073517 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:39.073817 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:39.073819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:39.073820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:39.073987 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:39.074058 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:39.074565 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 10:39:39.074664 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:39:39.075661 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.062287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43957 (local address 127.2.8.3:56207)
0924 10:39:39.062730 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:39.062775 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:39.064115 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:39.064889 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:39.064901 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:39.065473 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:39.066251 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:39.066273 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.067347 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.067370 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:39.068375 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:39.068385 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.068790 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.068837 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:39.069048 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.069662 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:39.069679 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:39.071240 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:39.073067 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:39.073072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:39.073085 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:39.073353 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:39.073598 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:39.073600 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:39.073601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:39.073730 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:39.074069 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:39.074074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:39.074353 (+ 279us) client_negotiation.cc:769] Sending connection context
0924 10:39:39.074511 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 10:39:39.074776 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":103,"threads_started":1}
I20260924 10:39:39.076849 3109 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43957
I20260924 10:39:39.077136 3109 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:39.077783 3109 heartbeater.cc:511] Master 127.2.8.62:43957 requested a full tablet report, sending...
I20260924 10:39:39.079023 2649 ts_manager.cc:194] Registered new tserver with Master: b0ef899ffff147fea9a0ddbb04b025e5 (127.2.8.3:33951)
I20260924 10:39:39.079643 2649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:56207
I20260924 10:39:39.091157 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:39:39.102553 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.094866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60872 (local address 127.2.8.62:43957)
0924 10:39:39.095027 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:39.095033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:39.095196 (+ 163us) server_negotiation.cc:407] Connection header received
0924 10:39:39.095350 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:39.095354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:39.095432 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:39.095526 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:39.096221 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.096733 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.097645 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.097861 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.097971 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.099356 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:39.099359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:39.099362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:39.099397 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:39.100848 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:39.101294 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:39.101299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:39.101301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:39.101388 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:39.101660 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:39.101665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:39.101666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:39.101884 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:39.102028 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:39.102259 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 10:39:39.102386 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:39:39.107669 2649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:39:39.110790 2649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:39:39.125507 3120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.119403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:33411 (local address 127.0.0.1:33052)
0924 10:39:39.119754 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:39.119797 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:39.120099 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:39.120709 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:39.120717 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:39.120755 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:39.120921 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:39.120929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.122566 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.122573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:39.123825 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:39.123834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.124898 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.124902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:39.125065 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.125112 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 10:39:39.125230 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 10:39:39.125345 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":87,"spinlock_wait_cycles":2176,"thread_start_us":56,"threads_started":1}
I20260924 10:39:39.125552 3121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.119568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.2.8.1:33411)
0924 10:39:39.120039 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:39.120053 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:39.120078 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:39:39.120280 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:39.120286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:39.120521 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:39.121094 (+ 573us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:39.121253 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.122416 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.123953 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.124753 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.124867 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.125061 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:39.125274 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 10:39:39.125370 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":143,"threads_started":1}
I20260924 10:39:39.125789 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.120417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.2.8.2:33607)
0924 10:39:39.120802 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:39.120815 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:39.121067 (+ 252us) server_negotiation.cc:407] Connection header received
0924 10:39:39.121220 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:39.121225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:39.121370 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:39.121530 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:39.121779 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.122913 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.124313 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.124924 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.125325 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.125470 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:39.125523 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 10:39:39.125616 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":104,"threads_started":1}
I20260924 10:39:39.125910 3122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.119618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.2.8.3:33951)
0924 10:39:39.120439 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:39.120454 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:39.120676 (+ 222us) server_negotiation.cc:407] Connection header received
0924 10:39:39.120924 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:39.120938 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:39.121162 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:39.121311 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:39.121744 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.122473 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.124147 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.124932 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.125053 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.125171 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:39.125507 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 10:39:39.125792 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":126,"threads_started":1}
I20260924 10:39:39.126078 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.120167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:33607 (local address 127.0.0.1:57180)
0924 10:39:39.120947 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:39.120956 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:39.121084 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:39.121523 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:39.121526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:39.121542 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:39.121670 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:39.121676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.123069 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.123072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:39.124200 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:39.124212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.125069 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.125072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:39.125143 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.125159 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 10:39:39.125206 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 10:39:39.125960 (+ 754us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":632,"thread_start_us":93,"threads_started":1}
I20260924 10:39:39.126334 3123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.119403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:33951 (local address 127.0.0.1:56664)
0924 10:39:39.120581 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:39.120594 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:39.120758 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:39.121458 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:39.121461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:39.121477 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:39.121581 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:39.121587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.122609 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.122612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:39.124026 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:39.124036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.125251 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.125254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:39.125355 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.125376 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:39:39.126119 (+ 743us) client_negotiation.cc:240] Negotiation successful
0924 10:39:39.126199 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"mutex_wait_us":512,"spinlock_wait_cycles":9216,"thread_start_us":116,"threads_started":1}
I20260924 10:39:39.128859 3043 tablet_service.cc:1467] Processing CreateTablet for tablet cc098d7f044d42e684944dad95bd4321 (DEFAULT_TABLE table=test-table [id=846e037ee97741d68874f2403cbaed99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:39.129025 2909 tablet_service.cc:1467] Processing CreateTablet for tablet cc098d7f044d42e684944dad95bd4321 (DEFAULT_TABLE table=test-table [id=846e037ee97741d68874f2403cbaed99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:39.129025 2775 tablet_service.cc:1467] Processing CreateTablet for tablet cc098d7f044d42e684944dad95bd4321 (DEFAULT_TABLE table=test-table [id=846e037ee97741d68874f2403cbaed99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:39.129988 3043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc098d7f044d42e684944dad95bd4321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:39.130012 2909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc098d7f044d42e684944dad95bd4321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:39.130010 2775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc098d7f044d42e684944dad95bd4321. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:39.136197 3126 tablet_bootstrap.cc:492] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5: Bootstrap starting.
I20260924 10:39:39.137718 3128 tablet_bootstrap.cc:492] T cc098d7f044d42e684944dad95bd4321 P 6793fd8ebedf4af8a027bdcd616f1d1d: Bootstrap starting.
I20260924 10:39:39.138664 3126 tablet_bootstrap.cc:654] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:39.139742 3126 log.cc:824] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:39.139788 3128 tablet_bootstrap.cc:654] T cc098d7f044d42e684944dad95bd4321 P 6793fd8ebedf4af8a027bdcd616f1d1d: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:39.140369 3127 tablet_bootstrap.cc:492] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5: Bootstrap starting.
I20260924 10:39:39.140606 3128 log.cc:824] T cc098d7f044d42e684944dad95bd4321 P 6793fd8ebedf4af8a027bdcd616f1d1d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:39.141847 3126 tablet_bootstrap.cc:492] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5: No bootstrap required, opened a new log
I20260924 10:39:39.142066 3126 ts_tablet_manager.cc:1397] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:39:39.142314 3127 tablet_bootstrap.cc:654] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:39.142652 3128 tablet_bootstrap.cc:492] T cc098d7f044d42e684944dad95bd4321 P 6793fd8ebedf4af8a027bdcd616f1d1d: No bootstrap required, opened a new log
I20260924 10:39:39.142822 3128 ts_tablet_manager.cc:1397] T cc098d7f044d42e684944dad95bd4321 P 6793fd8ebedf4af8a027bdcd616f1d1d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:39:39.143567 3127 log.cc:824] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:39.145144 3127 tablet_bootstrap.cc:492] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5: No bootstrap required, opened a new log
I20260924 10:39:39.145328 3127 ts_tablet_manager.cc:1397] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:39:39.145937 3126 raft_consensus.cc:348] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e95a887b02e34681bc7cb30b03bd18e5" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33411 } } peers { permanent_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 33951 } } peers { permanent_uuid: "6793fd8ebedf4af8a027bdcd616f1d1d" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33607 } }
I20260924 10:39:39.146229 3126 raft_consensus.cc:374] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:39.146286 3126 raft_consensus.cc:729] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ef899ffff147fea9a0ddbb04b025e5, State: Initialized, Role: FOLLOWER
I20260924 10:39:39.146600 3128 raft_consensus.cc:348] T cc098d7f044d42e684944dad95bd4321 P 6793fd8ebedf4af8a027bdcd616f1d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e95a887b02e34681bc7cb30b03bd18e5" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33411 } } peers { permanent_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 33951 } } peers { permanent_uuid: "6793fd8ebedf4af8a027bdcd616f1d1d" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33607 } }
I20260924 10:39:39.146714 3126 consensus_queue.cc:260] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e95a887b02e34681bc7cb30b03bd18e5" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33411 } } peers { permanent_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 33951 } } peers { permanent_uuid: "6793fd8ebedf4af8a027bdcd616f1d1d" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33607 } }
I20260924 10:39:39.146848 3128 raft_consensus.cc:374] T cc098d7f044d42e684944dad95bd4321 P 6793fd8ebedf4af8a027bdcd616f1d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:39.146907 3128 raft_consensus.cc:729] T cc098d7f044d42e684944dad95bd4321 P 6793fd8ebedf4af8a027bdcd616f1d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6793fd8ebedf4af8a027bdcd616f1d1d, State: Initialized, Role: FOLLOWER
I20260924 10:39:39.147343 3109 heartbeater.cc:503] Master 127.2.8.62:43957 was elected leader, sending a full tablet report...
I20260924 10:39:39.147348 3128 consensus_queue.cc:260] T cc098d7f044d42e684944dad95bd4321 P 6793fd8ebedf4af8a027bdcd616f1d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e95a887b02e34681bc7cb30b03bd18e5" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33411 } } peers { permanent_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 33951 } } peers { permanent_uuid: "6793fd8ebedf4af8a027bdcd616f1d1d" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33607 } }
I20260924 10:39:39.148360 2975 heartbeater.cc:503] Master 127.2.8.62:43957 was elected leader, sending a full tablet report...
I20260924 10:39:39.148476 3128 ts_tablet_manager.cc:1428] T cc098d7f044d42e684944dad95bd4321 P 6793fd8ebedf4af8a027bdcd616f1d1d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:39:39.148726 3127 raft_consensus.cc:348] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e95a887b02e34681bc7cb30b03bd18e5" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33411 } } peers { permanent_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 33951 } } peers { permanent_uuid: "6793fd8ebedf4af8a027bdcd616f1d1d" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33607 } }
I20260924 10:39:39.149012 3127 raft_consensus.cc:374] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:39.149068 3127 raft_consensus.cc:729] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e95a887b02e34681bc7cb30b03bd18e5, State: Initialized, Role: FOLLOWER
I20260924 10:39:39.149438 3127 consensus_queue.cc:260] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e95a887b02e34681bc7cb30b03bd18e5" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33411 } } peers { permanent_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 33951 } } peers { permanent_uuid: "6793fd8ebedf4af8a027bdcd616f1d1d" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33607 } }
I20260924 10:39:39.149633 3126 ts_tablet_manager.cc:1428] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260924 10:39:39.150072 2841 heartbeater.cc:503] Master 127.2.8.62:43957 was elected leader, sending a full tablet report...
I20260924 10:39:39.150338 3127 ts_tablet_manager.cc:1428] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260924 10:39:39.310881 3110 tablet.cc:2369] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:39:39.341290 3134 raft_consensus.cc:482] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:39:39.341405 3132 raft_consensus.cc:482] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:39:39.341547 3134 raft_consensus.cc:504] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e95a887b02e34681bc7cb30b03bd18e5" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33411 } } peers { permanent_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 33951 } } peers { permanent_uuid: "6793fd8ebedf4af8a027bdcd616f1d1d" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33607 } }
I20260924 10:39:39.341547 3132 raft_consensus.cc:504] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e95a887b02e34681bc7cb30b03bd18e5" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33411 } } peers { permanent_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 33951 } } peers { permanent_uuid: "6793fd8ebedf4af8a027bdcd616f1d1d" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33607 } }
I20260924 10:39:39.342904 3132 leader_election.cc:288] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e95a887b02e34681bc7cb30b03bd18e5 (127.2.8.1:33411), 6793fd8ebedf4af8a027bdcd616f1d1d (127.2.8.2:33607)
I20260924 10:39:39.343828 3134 leader_election.cc:288] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0ef899ffff147fea9a0ddbb04b025e5 (127.2.8.3:33951), 6793fd8ebedf4af8a027bdcd616f1d1d (127.2.8.2:33607)
I20260924 10:39:39.347705 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.343149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:33411 (local address 127.2.8.3:42429)
0924 10:39:39.343453 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:39.343470 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:39.343567 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:39.344008 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:39.344011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:39.344028 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:39.344152 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:39.344159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.345140 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.345143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:39.346340 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:39.346350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.347317 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.347320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:39.347422 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.347443 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:39:39.347507 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 10:39:39.347571 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260924 10:39:39.348165 3121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.343388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:42429 (local address 127.2.8.1:33411)
0924 10:39:39.343709 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:39.343713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:39.343725 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:39:39.343768 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:39.343771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:39.343817 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:39.343889 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:39.344287 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.345027 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.346469 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.347087 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.347196 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.347257 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:39.347965 (+ 708us) server_negotiation.cc:299] Negotiation successful
0924 10:39:39.348040 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":264}
I20260924 10:39:39.348266 3122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.344002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:53479 (local address 127.2.8.3:33951)
0924 10:39:39.344140 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:39.344143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:39.344154 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:39:39.344188 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:39.344194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:39.344238 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:39.344299 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:39.344847 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.345587 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.347207 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.347723 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.347834 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.347897 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:39.348088 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 10:39:39.348156 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260924 10:39:39.348372 2976 tablet.cc:2369] T cc098d7f044d42e684944dad95bd4321 P 6793fd8ebedf4af8a027bdcd616f1d1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:39:39.348718 2795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc098d7f044d42e684944dad95bd4321" candidate_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e95a887b02e34681bc7cb30b03bd18e5" is_pre_election: true
I20260924 10:39:39.348856 3125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.343624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:36561 (local address 127.2.8.2:33607)
0924 10:39:39.343773 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:39.343778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:39.344175 (+ 397us) server_negotiation.cc:407] Connection header received
0924 10:39:39.344340 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:39.344342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:39.344404 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:39.344455 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:39.344864 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.345750 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.347281 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.348050 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.348139 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.348190 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:39.348652 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 10:39:39.348728 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":117,"server-negotiator.queue_time_us":43,"spinlock_wait_cycles":9216}
I20260924 10:39:39.348945 2795 raft_consensus.cc:2454] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ef899ffff147fea9a0ddbb04b025e5 in term 0.
I20260924 10:39:39.349319 3136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.343216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:33607 (local address 127.2.8.1:36561)
0924 10:39:39.344089 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:39.344104 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:39.344240 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:39.344589 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:39.344591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:39.344609 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:39.344741 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:39.344747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.345887 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.345890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:39.347143 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:39.347152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.348139 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.348143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:39.348508 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.348535 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 10:39:39.349109 (+ 574us) client_negotiation.cc:240] Negotiation successful
0924 10:39:39.349178 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":110,"threads_started":1}
I20260924 10:39:39.349329 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.343216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:33607 (local address 127.2.8.3:33753)
0924 10:39:39.343611 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:39.343624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:39.343720 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:39.344516 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:39.344519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:39.344538 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:39.344652 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:39.344658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.345882 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.345885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:39.347142 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:39.347152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.348881 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.348886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:39.349044 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.349072 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:39:39.349123 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 10:39:39.349186 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":123,"threads_started":1}
I20260924 10:39:39.349457 2999 leader_election.cc:302] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0ef899ffff147fea9a0ddbb04b025e5, e95a887b02e34681bc7cb30b03bd18e5; no voters:
I20260924 10:39:39.349830 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.343740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:33753 (local address 127.2.8.2:33607)
0924 10:39:39.344220 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:39.344231 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:39.344241 (+ 10us) server_negotiation.cc:407] Connection header received
0924 10:39:39.344312 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:39.344318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:39.344404 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:39.344750 (+ 346us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:39.344851 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.345750 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.347288 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.348724 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.349496 (+ 772us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.349566 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:39.349609 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 10:39:39.349671 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":412,"spinlock_wait_cycles":27648,"thread_start_us":73,"threads_started":1}
I20260924 10:39:39.349782 3132 raft_consensus.cc:2781] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:39:39.349871 3132 raft_consensus.cc:482] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:39:39.349892 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.343150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:33951 (local address 127.2.8.1:53479)
0924 10:39:39.343585 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:39.343599 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:39.343847 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:39.344554 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:39.344557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:39.344573 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:39.344710 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:39.344716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.345716 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.345719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:39.347087 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:39.347094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.347835 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.347838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:39.347936 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.347960 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:39:39.348383 (+ 423us) client_negotiation.cc:240] Negotiation successful
0924 10:39:39.349755 (+ 1372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":1445,"thread_start_us":102,"threads_started":1}
I20260924 10:39:39.349848 2929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc098d7f044d42e684944dad95bd4321" candidate_uuid: "e95a887b02e34681bc7cb30b03bd18e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6793fd8ebedf4af8a027bdcd616f1d1d" is_pre_election: true
I20260924 10:39:39.349922 3132 raft_consensus.cc:3037] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:39.350126 2929 raft_consensus.cc:2454] T cc098d7f044d42e684944dad95bd4321 P 6793fd8ebedf4af8a027bdcd616f1d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e95a887b02e34681bc7cb30b03bd18e5 in term 0.
I20260924 10:39:39.350185 2928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc098d7f044d42e684944dad95bd4321" candidate_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6793fd8ebedf4af8a027bdcd616f1d1d" is_pre_election: true
I20260924 10:39:39.350286 2928 raft_consensus.cc:2454] T cc098d7f044d42e684944dad95bd4321 P 6793fd8ebedf4af8a027bdcd616f1d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ef899ffff147fea9a0ddbb04b025e5 in term 0.
I20260924 10:39:39.350497 3063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc098d7f044d42e684944dad95bd4321" candidate_uuid: "e95a887b02e34681bc7cb30b03bd18e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" is_pre_election: true
I20260924 10:39:39.350553 2728 leader_election.cc:302] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6793fd8ebedf4af8a027bdcd616f1d1d, e95a887b02e34681bc7cb30b03bd18e5; no voters:
I20260924 10:39:39.350783 3134 raft_consensus.cc:2781] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:39:39.350855 3134 raft_consensus.cc:482] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:39:39.350885 3134 raft_consensus.cc:3037] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:39.351227 3132 raft_consensus.cc:504] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e95a887b02e34681bc7cb30b03bd18e5" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33411 } } peers { permanent_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 33951 } } peers { permanent_uuid: "6793fd8ebedf4af8a027bdcd616f1d1d" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33607 } }
I20260924 10:39:39.351485 3063 raft_consensus.cc:2379] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e95a887b02e34681bc7cb30b03bd18e5 in current term 1: Already voted for candidate b0ef899ffff147fea9a0ddbb04b025e5 in this term.
I20260924 10:39:39.351605 3132 leader_election.cc:288] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [CANDIDATE]: Term 1 election: Requested vote from peers e95a887b02e34681bc7cb30b03bd18e5 (127.2.8.1:33411), 6793fd8ebedf4af8a027bdcd616f1d1d (127.2.8.2:33607)
I20260924 10:39:39.351958 2928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc098d7f044d42e684944dad95bd4321" candidate_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6793fd8ebedf4af8a027bdcd616f1d1d"
I20260924 10:39:39.351958 2795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc098d7f044d42e684944dad95bd4321" candidate_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e95a887b02e34681bc7cb30b03bd18e5"
I20260924 10:39:39.352065 2928 raft_consensus.cc:3037] T cc098d7f044d42e684944dad95bd4321 P 6793fd8ebedf4af8a027bdcd616f1d1d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:39.352120 3134 raft_consensus.cc:504] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e95a887b02e34681bc7cb30b03bd18e5" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33411 } } peers { permanent_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 33951 } } peers { permanent_uuid: "6793fd8ebedf4af8a027bdcd616f1d1d" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33607 } }
I20260924 10:39:39.352430 2795 raft_consensus.cc:2379] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b0ef899ffff147fea9a0ddbb04b025e5 in current term 1: Already voted for candidate e95a887b02e34681bc7cb30b03bd18e5 in this term.
I20260924 10:39:39.352504 3134 leader_election.cc:288] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [CANDIDATE]: Term 1 election: Requested vote from peers b0ef899ffff147fea9a0ddbb04b025e5 (127.2.8.3:33951), 6793fd8ebedf4af8a027bdcd616f1d1d (127.2.8.2:33607)
I20260924 10:39:39.352804 2929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc098d7f044d42e684944dad95bd4321" candidate_uuid: "e95a887b02e34681bc7cb30b03bd18e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6793fd8ebedf4af8a027bdcd616f1d1d"
I20260924 10:39:39.353385 2928 raft_consensus.cc:2454] T cc098d7f044d42e684944dad95bd4321 P 6793fd8ebedf4af8a027bdcd616f1d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0ef899ffff147fea9a0ddbb04b025e5 in term 1.
I20260924 10:39:39.353662 2996 leader_election.cc:302] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6793fd8ebedf4af8a027bdcd616f1d1d, b0ef899ffff147fea9a0ddbb04b025e5; no voters: e95a887b02e34681bc7cb30b03bd18e5
I20260924 10:39:39.353641 3063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc098d7f044d42e684944dad95bd4321" candidate_uuid: "e95a887b02e34681bc7cb30b03bd18e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0ef899ffff147fea9a0ddbb04b025e5"
I20260924 10:39:39.353839 3063 raft_consensus.cc:2379] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e95a887b02e34681bc7cb30b03bd18e5 in current term 1: Already voted for candidate b0ef899ffff147fea9a0ddbb04b025e5 in this term.
I20260924 10:39:39.354030 3132 raft_consensus.cc:2781] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:39.354187 2729 leader_election.cc:302] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e95a887b02e34681bc7cb30b03bd18e5; no voters: 6793fd8ebedf4af8a027bdcd616f1d1d, b0ef899ffff147fea9a0ddbb04b025e5
I20260924 10:39:39.354260 3132 raft_consensus.cc:686] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [term 1 LEADER]: Becoming Leader. State: Replica: b0ef899ffff147fea9a0ddbb04b025e5, State: Running, Role: LEADER
I20260924 10:39:39.354403 3134 raft_consensus.cc:2726] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 10:39:39.354589 3132 consensus_queue.cc:237] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e95a887b02e34681bc7cb30b03bd18e5" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33411 } } peers { permanent_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 33951 } } peers { permanent_uuid: "6793fd8ebedf4af8a027bdcd616f1d1d" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33607 } }
I20260924 10:39:39.357564 2649 catalog_manager.cc:5495] T cc098d7f044d42e684944dad95bd4321 P b0ef899ffff147fea9a0ddbb04b025e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0ef899ffff147fea9a0ddbb04b025e5 (127.2.8.3). New cstate: current_term: 1 leader_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e95a887b02e34681bc7cb30b03bd18e5" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0ef899ffff147fea9a0ddbb04b025e5" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 33951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6793fd8ebedf4af8a027bdcd616f1d1d" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33607 } health_report { overall_health: UNKNOWN } } }
W20260924 10:39:39.382711 2842 tablet.cc:2369] T cc098d7f044d42e684944dad95bd4321 P e95a887b02e34681bc7cb30b03bd18e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:39:39 dist-test-slave-j38h krb5kdc[2605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.1@KRBTEST.COM
I20260924 10:39:39.423223 3121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.414413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.2.8.1:33411)
0924 10:39:39.414569 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:39.414573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:39.414770 (+ 197us) server_negotiation.cc:407] Connection header received
0924 10:39:39.414906 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:39.414909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:39.414953 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:39.415017 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:39.415792 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.416289 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.417314 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.417553 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.417641 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.420045 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:39.420058 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:39.420072 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:39.420094 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:39.421814 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:39.422184 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:39.422189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:39.422190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:39.422262 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:39.422488 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:39.422490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:39.422491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:39.422715 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:39.422866 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:39.423013 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 10:39:39.423101 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 10:39:39 dist-test-slave-j38h krb5kdc[2605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246379, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:39:39 dist-test-slave-j38h krb5kdc[2605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246379, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.8.1@KRBTEST.COM
I20260924 10:39:39.450555 3121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.441355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.2.8.1:33411)
0924 10:39:39.441522 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:39.441527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:39.441660 (+ 133us) server_negotiation.cc:407] Connection header received
0924 10:39:39.441835 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:39.441838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:39.441887 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:39.441975 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:39.442662 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.443208 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.444423 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.444775 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.444864 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.447435 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:39.447438 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:39.447440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:39.447463 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:39.448824 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:39.449286 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:39.449290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:39.449292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:39.449379 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:39.449658 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:39.449661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:39.449663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:39.449825 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:39.449962 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:39.450246 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 10:39:39.450384 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 10:39:39 dist-test-slave-j38h krb5kdc[2605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:39:39 dist-test-slave-j38h krb5kdc[2605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.1@KRBTEST.COM
I20260924 10:39:39.475117 3121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:39.467408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33082 (local address 127.2.8.1:33411)
0924 10:39:39.467584 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:39.467591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:39.467666 (+ 75us) server_negotiation.cc:407] Connection header received
0924 10:39:39.467770 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:39.467773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:39.467814 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:39.467893 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:39.468514 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.469085 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.469842 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:39.470046 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:39.470116 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:39.472438 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:39.472441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:39.472443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:39.472464 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:39.473901 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:39.474243 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:39.474245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:39.474246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:39.474312 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:39.474521 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:39.474523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:39.474524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:39.474664 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:39.474757 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:39.474920 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 10:39:39.475001 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260924 10:39:39.476643 2755 tablet_service.cc:2908] Rejecting scan request for tablet cc098d7f044d42e684944dad95bd4321: Uninitialized: clean time has not yet been initialized
I20260924 10:39:39.478580 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 2712
I20260924 10:39:39.483906 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 2846
I20260924 10:39:39.489284 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 2980
I20260924 10:39:39.495293 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 2618
2026-09-24T10:39:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2458 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:39 dist-test-slave-j38h krb5kdc[3153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:39 dist-test-slave-j38h krb5kdc[3153](info): set up 2 sockets
Sep 24 10:39:39 dist-test-slave-j38h krb5kdc[3153](info): commencing operation
krb5kdc: starting...
W20260924 10:39:40.556867 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:39:40 dist-test-slave-j38h krb5kdc[3153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:39:40.676465 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:42661
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:42661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:40.788166 3166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:40.788480 3166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:40.788550 3166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:40.792559 3166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:40.792654 3166 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 10:39:40.792686 3166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:40.792730 3166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:40.792762 3166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:40.792796 3166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:40.798270 3166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:42661
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:42661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:40.799407 3166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:40.800511 3166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:40.809376 3174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:40.809476 3166 server_base.cc:1034] running on GCE node
W20260924 10:39:40.810176 3172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:40.810452 3171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:40.811054 3166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:40.812158 3166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:40.813331 3166 hybrid_clock.cc:648] HybridClock initialized: now 1790246380813320 us; error 28 us; skew 500 ppm
Sep 24 10:39:40 dist-test-slave-j38h krb5kdc[3153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246380, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:40.816079 3166 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:39:40.817059 3166 webserver.cc:460] Webserver started at http://127.2.8.62:33523/ using document root <none> and password file <none>
I20260924 10:39:40.817654 3166 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:40.817747 3166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:40.817986 3166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:40.820032 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "89e89066249843bca8b9b843d4b0a443"
format_stamp: "Formatted at 2026-09-24 10:39:40 on dist-test-slave-j38h"
server_key: "0ec98c5008a627370a4dc054f08a2b12"
server_key_iv: "d0a65d4b6e79a90a9dd6710a55c7e710"
server_key_version: "encryptionkey@0"
I20260924 10:39:40.820587 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "89e89066249843bca8b9b843d4b0a443"
format_stamp: "Formatted at 2026-09-24 10:39:40 on dist-test-slave-j38h"
server_key: "0ec98c5008a627370a4dc054f08a2b12"
server_key_iv: "d0a65d4b6e79a90a9dd6710a55c7e710"
server_key_version: "encryptionkey@0"
I20260924 10:39:40.824239 3166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:39:40.826678 3181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:40.827833 3166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:39:40.827972 3166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "89e89066249843bca8b9b843d4b0a443"
format_stamp: "Formatted at 2026-09-24 10:39:40 on dist-test-slave-j38h"
server_key: "0ec98c5008a627370a4dc054f08a2b12"
server_key_iv: "d0a65d4b6e79a90a9dd6710a55c7e710"
server_key_version: "encryptionkey@0"
I20260924 10:39:40.828079 3166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:40.842299 3166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:40.845734 3166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:40.845958 3166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:40.854993 3166 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:42661
I20260924 10:39:40.855048 3243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:42661 every 8 connection(s)
I20260924 10:39:40.856418 3166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:40.860307 3244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:40.862836 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 3166
I20260924 10:39:40.862929 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:39:40.863201 2080 external_mini_cluster.cc:1428] Setting key 24e3a67a228c0d1d2067ea7edaa00138
I20260924 10:39:40.866704 3244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443: Bootstrap starting.
I20260924 10:39:40.868891 3244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:40.869632 3244 log.cc:824] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:40.871938 3244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443: No bootstrap required, opened a new log
Sep 24 10:39:40 dist-test-slave-j38h krb5kdc[3153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:40.875279 3244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e89066249843bca8b9b843d4b0a443" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42661 } }
I20260924 10:39:40.875527 3244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:40.875615 3244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89e89066249843bca8b9b843d4b0a443, State: Initialized, Role: FOLLOWER
I20260924 10:39:40.876096 3244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e89066249843bca8b9b843d4b0a443" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42661 } }
I20260924 10:39:40.876245 3244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:40.876320 3244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:40.876422 3244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:40.877415 3244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e89066249843bca8b9b843d4b0a443" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42661 } }
I20260924 10:39:40.877568 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:40.864644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32810 (local address 127.2.8.62:42661)
0924 10:39:40.865163 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:40.865178 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:40.865215 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:39:40.865904 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:40.865921 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:40.866269 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:40.866610 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:40.867397 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:40.868368 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:40.869306 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:40.869811 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:40.869963 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:40.872698 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:40.872704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:40.872718 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:40.872743 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:40.874523 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:40.874964 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:40.874974 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:40.874979 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:40.875083 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:40.875369 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:40.875373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:40.875374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:40.875865 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:40.876071 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:40.876315 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 10:39:40.876729 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":140,"threads_started":1}
I20260924 10:39:40.877812 3244 leader_election.cc:302] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89e89066249843bca8b9b843d4b0a443; no voters:
I20260924 10:39:40.878131 3244 leader_election.cc:288] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:40.878245 3249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:40.878446 3249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [term 1 LEADER]: Becoming Leader. State: Replica: 89e89066249843bca8b9b843d4b0a443, State: Running, Role: LEADER
I20260924 10:39:40.878732 3249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e89066249843bca8b9b843d4b0a443" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42661 } }
I20260924 10:39:40.879366 3244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:40.880394 3250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89e89066249843bca8b9b843d4b0a443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e89066249843bca8b9b843d4b0a443" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42661 } } }
I20260924 10:39:40.880438 3251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89e89066249843bca8b9b843d4b0a443. Latest consensus state: current_term: 1 leader_uuid: "89e89066249843bca8b9b843d4b0a443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89e89066249843bca8b9b843d4b0a443" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42661 } } }
I20260924 10:39:40.880508 3251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:40.880508 3250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:40.880864 3258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:40.883973 3258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:40.889973 3258 catalog_manager.cc:1347] Generated new cluster ID: a5cc96fabc3241e1b98c94e4f8b47aa6
I20260924 10:39:40.890036 3258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:40.905987 3258 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:40.907209 3258 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:40.929430 3258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443: Generated new TSK 0
I20260924 10:39:40.930310 3258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:39:40.964218 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:42661
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:41.073309 3270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:41.073542 3270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:41.073585 3270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:41.077564 3270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:41.077615 3270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:41.077653 3270 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:39:41.077731 3270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:39:41.083199 3270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.8.62:42661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:41.084167 3270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:41.085209 3270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:41.094551 3275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:41.094551 3276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:41.094771 3270 server_base.cc:1034] running on GCE node
W20260924 10:39:41.094563 3278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:41.095381 3270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:41.096061 3270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:41.097239 3270 hybrid_clock.cc:648] HybridClock initialized: now 1790246381097214 us; error 43 us; skew 500 ppm
Sep 24 10:39:41 dist-test-slave-j38h krb5kdc[3153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246381, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:41.099922 3270 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:39:41.100683 3270 webserver.cc:460] Webserver started at http://127.2.8.1:40107/ using document root <none> and password file <none>
I20260924 10:39:41.101243 3270 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:41.101302 3270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:41.101459 3270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:41.103331 3270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d"
format_stamp: "Formatted at 2026-09-24 10:39:41 on dist-test-slave-j38h"
server_key: "e4b1700aff166509bbd95bd17c827a6d"
server_key_iv: "03b28344a9ed7a8234cbdecf5cbac402"
server_key_version: "encryptionkey@0"
I20260924 10:39:41.103814 3270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d"
format_stamp: "Formatted at 2026-09-24 10:39:41 on dist-test-slave-j38h"
server_key: "e4b1700aff166509bbd95bd17c827a6d"
server_key_iv: "03b28344a9ed7a8234cbdecf5cbac402"
server_key_version: "encryptionkey@0"
I20260924 10:39:41.107373 3270 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:39:41.109663 3285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:41.110853 3270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:39:41.110978 3270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d"
format_stamp: "Formatted at 2026-09-24 10:39:41 on dist-test-slave-j38h"
server_key: "e4b1700aff166509bbd95bd17c827a6d"
server_key_iv: "03b28344a9ed7a8234cbdecf5cbac402"
server_key_version: "encryptionkey@0"
I20260924 10:39:41.111080 3270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:41.132082 3270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:41.134987 3270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:41.135181 3270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:41.136524 3270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:41.136597 3270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:41.136667 3270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:41.136720 3270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:39:41 dist-test-slave-j38h krb5kdc[3153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246381, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:41.147436 3270 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:35811
I20260924 10:39:41.147457 3400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:35811 every 8 connection(s)
I20260924 10:39:41.148650 3270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:39:41.150197 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.137489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:35299 (local address 127.2.8.62:42661)
0924 10:39:41.137669 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.137676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.138457 (+ 781us) server_negotiation.cc:407] Connection header received
0924 10:39:41.139487 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.139492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.139557 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.139605 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:41.141310 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.141938 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.143066 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.143335 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.143477 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.146437 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:41.146440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:41.146443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:41.146471 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:41.148133 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:41.148649 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:41.148652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:41.148654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:41.148742 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:41.149091 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:41.149096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:41.149098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:41.149324 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:41.149421 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.149969 (+ 548us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.150081 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 10:39:41.150558 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 3270
I20260924 10:39:41.150665 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:39:41.150913 2080 external_mini_cluster.cc:1428] Setting key ce9b5a20d53c4f2391f371fb56a85047
I20260924 10:39:41.151075 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.137858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:42661 (local address 127.2.8.1:35299)
0924 10:39:41.138291 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:41.138338 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:41.139236 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:41.140028 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:41.140037 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:41.140488 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:41.141167 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.141183 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.142063 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.142066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.142940 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.142949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.143466 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.143527 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.143760 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.144392 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:41.144416 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:41.146269 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:41.148250 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:41.148255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:41.148270 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:41.148530 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:41.148817 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:41.148822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:41.148824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:41.148963 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:41.149472 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:41.149477 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:41.149735 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 10:39:41.149880 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 10:39:41.150117 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":120,"threads_started":1}
I20260924 10:39:41.152281 3401 heartbeater.cc:347] Connected to a master server at 127.2.8.62:42661
I20260924 10:39:41.152411 3401 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:41.152860 3401 heartbeater.cc:511] Master 127.2.8.62:42661 requested a full tablet report, sending...
I20260924 10:39:41.154470 3198 ts_manager.cc:194] Registered new tserver with Master: 2cb8d14c65eb4fd8aa0df98b3ed9034d (127.2.8.1:35811)
I20260924 10:39:41.156060 3198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:35299
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
I20260924 10:39:41.161638 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.154563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:49465 (local address 127.2.8.62:42661)
0924 10:39:41.154689 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.154693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.154915 (+ 222us) server_negotiation.cc:407] Connection header received
0924 10:39:41.155024 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.155027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.155066 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.155155 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:41.155560 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.156065 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.157042 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.157268 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.157379 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.158539 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:41.158542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:41.158545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:41.158565 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:41.160029 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:41.160454 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:41.160459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:41.160461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:41.160536 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:41.160812 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:41.160815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:41.160816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:41.161038 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:41.161146 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.161395 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.161493 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:39:41.161671 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.154527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:42661 (local address 127.2.8.1:49465)
0924 10:39:41.154835 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:41.154847 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:41.154950 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:41.155180 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:41.155182 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:41.155368 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:41.155454 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.155459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.156186 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.156189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.156882 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.156890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.157369 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.157386 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.157474 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.158094 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:41.158104 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:41.158410 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:41.160158 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:41.160161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:41.160163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:41.160357 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:41.160625 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:41.160630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:41.160632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:41.160716 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:41.161146 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:41.161150 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:41.161238 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 10:39:41.161354 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 10:39:41.161486 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":64,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:39:41.177963 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:42661
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:41.285703 3410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:41.285953 3410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:41.285996 3410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:41.289954 3410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:41.290007 3410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:41.290045 3410 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:39:41.290114 3410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:39:41.295246 3410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.8.62:42661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:41.296244 3410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:41.297253 3410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:39:41.306751 3410 server_base.cc:1034] running on GCE node
W20260924 10:39:41.306653 3415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:41.306639 3416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:41.306636 3418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:41.307303 3410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:41.307858 3410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:41.309068 3410 hybrid_clock.cc:648] HybridClock initialized: now 1790246381309035 us; error 44 us; skew 500 ppm
Sep 24 10:39:41 dist-test-slave-j38h krb5kdc[3153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246381, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:41.311923 3410 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:39:41.312788 3410 webserver.cc:460] Webserver started at http://127.2.8.2:38691/ using document root <none> and password file <none>
I20260924 10:39:41.313395 3410 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:41.313485 3410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:41.313726 3410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:41.315755 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "06de1d796776422abb72a1cd78b2bfc7"
format_stamp: "Formatted at 2026-09-24 10:39:41 on dist-test-slave-j38h"
server_key: "bb83074ece4b11ec02c04ab4253472c8"
server_key_iv: "0fac10a0db7cdff9a18c4be0f7d72070"
server_key_version: "encryptionkey@0"
I20260924 10:39:41.316305 3410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "06de1d796776422abb72a1cd78b2bfc7"
format_stamp: "Formatted at 2026-09-24 10:39:41 on dist-test-slave-j38h"
server_key: "bb83074ece4b11ec02c04ab4253472c8"
server_key_iv: "0fac10a0db7cdff9a18c4be0f7d72070"
server_key_version: "encryptionkey@0"
I20260924 10:39:41.319902 3410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:39:41.322237 3425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:41.323379 3410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:39:41.323504 3410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "06de1d796776422abb72a1cd78b2bfc7"
format_stamp: "Formatted at 2026-09-24 10:39:41 on dist-test-slave-j38h"
server_key: "bb83074ece4b11ec02c04ab4253472c8"
server_key_iv: "0fac10a0db7cdff9a18c4be0f7d72070"
server_key_version: "encryptionkey@0"
I20260924 10:39:41.323606 3410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:41.345228 3410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:41.348608 3410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:41.348816 3410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:41.350279 3410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:41.350353 3410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:41.350419 3410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:41.350471 3410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:39:41 dist-test-slave-j38h krb5kdc[3153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246381, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:41.361513 3410 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:33291
I20260924 10:39:41.361588 3540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:33291 every 8 connection(s)
I20260924 10:39:41.362434 3410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:39:41.363560 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 3410
I20260924 10:39:41.363692 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:39:41.363989 2080 external_mini_cluster.cc:1428] Setting key 91a92d64e4613bc628ea609e0f1e58e2
I20260924 10:39:41.365468 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.351199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:45199 (local address 127.2.8.62:42661)
0924 10:39:41.351322 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.351326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.352402 (+ 1076us) server_negotiation.cc:407] Connection header received
0924 10:39:41.353661 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.353666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.353732 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.353849 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:41.355338 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.356249 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.358270 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.358623 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.358752 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.361338 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:41.361349 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:41.361353 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:41.361378 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:41.363099 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:41.363722 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:41.363725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:41.363727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:41.363823 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:41.364168 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:41.364172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:41.364173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:41.364371 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:41.364489 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.365099 (+ 610us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.365233 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:39:41.366182 3434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.351767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:42661 (local address 127.2.8.2:45199)
0924 10:39:41.352244 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:41.352285 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:41.353429 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:41.354058 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:41.354066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:41.354535 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:41.355178 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.355215 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.356410 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.356414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.358112 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.358123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.358754 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.358800 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.358948 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.359589 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:41.359607 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:41.361143 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:41.363229 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:41.363243 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:41.363254 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:41.363597 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:41.363921 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:41.363924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:41.363926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:41.364062 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:41.364485 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:41.364491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:41.364774 (+ 283us) client_negotiation.cc:769] Sending connection context
0924 10:39:41.365000 (+ 226us) client_negotiation.cc:240] Negotiation successful
0924 10:39:41.365356 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":125,"threads_started":1}
I20260924 10:39:41.367332 3541 heartbeater.cc:347] Connected to a master server at 127.2.8.62:42661
I20260924 10:39:41.367488 3541 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:41.368074 3541 heartbeater.cc:511] Master 127.2.8.62:42661 requested a full tablet report, sending...
I20260924 10:39:41.369174 3196 ts_manager.cc:194] Registered new tserver with Master: 06de1d796776422abb72a1cd78b2bfc7 (127.2.8.2:33291)
I20260924 10:39:41.370184 3196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:45199
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
I20260924 10:39:41.380303 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.369555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:42661 (local address 127.2.8.2:33369)
0924 10:39:41.373158 (+ 3603us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:41.373170 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:41.373249 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:41.373485 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:41.373487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:41.373648 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:41.373741 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.373746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.374826 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.374828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.375377 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.375386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.375784 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.375798 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.375865 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.376395 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:41.376407 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:41.376776 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:41.378514 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:41.378520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:41.378526 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:41.378798 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:41.379112 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:41.379114 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:41.379115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:41.379220 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:41.379748 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:41.379750 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:41.379823 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 10:39:41.379978 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 10:39:41.380110 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3532,"thread_start_us":94,"threads_started":1}
I20260924 10:39:41.380343 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.369660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:33369 (local address 127.2.8.62:42661)
0924 10:39:41.369764 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.369768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.373247 (+ 3479us) server_negotiation.cc:407] Connection header received
0924 10:39:41.373327 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.373330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.373376 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.373466 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:41.373858 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.374708 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.375494 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.375699 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.375779 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.376909 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:41.376912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:41.376915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:41.376943 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:41.378406 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:41.378910 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:41.378916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:41.378918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:41.379019 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:41.379339 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:41.379345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:41.379348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:41.379641 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:41.379745 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.380067 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.380172 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:39:41.392341 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:42661
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:41.504912 3550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:41.505249 3550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:41.505331 3550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:41.509373 3550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:41.509445 3550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:41.509514 3550 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:39:41.509589 3550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:39:41.514837 3550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.8.62:42661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:41.515949 3550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:41.517019 3550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:41.526631 3556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:41.526715 3558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:41.526885 3550 server_base.cc:1034] running on GCE node
W20260924 10:39:41.526672 3555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:41.527283 3550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:41.527945 3550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:41.529109 3550 hybrid_clock.cc:648] HybridClock initialized: now 1790246381529070 us; error 50 us; skew 500 ppm
Sep 24 10:39:41 dist-test-slave-j38h krb5kdc[3153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246381, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:41.532214 3550 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:39:41.533181 3550 webserver.cc:460] Webserver started at http://127.2.8.3:46025/ using document root <none> and password file <none>
I20260924 10:39:41.533725 3550 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:41.533813 3550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:41.534034 3550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:41.535995 3550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "9553b930d3874fb5a9255cbc3742f7bd"
format_stamp: "Formatted at 2026-09-24 10:39:41 on dist-test-slave-j38h"
server_key: "84b367f36498a0c657b196e57abe8fa4"
server_key_iv: "4704105d480999aad6438f55df41ac4f"
server_key_version: "encryptionkey@0"
I20260924 10:39:41.536548 3550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "9553b930d3874fb5a9255cbc3742f7bd"
format_stamp: "Formatted at 2026-09-24 10:39:41 on dist-test-slave-j38h"
server_key: "84b367f36498a0c657b196e57abe8fa4"
server_key_iv: "4704105d480999aad6438f55df41ac4f"
server_key_version: "encryptionkey@0"
I20260924 10:39:41.540045 3550 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:39:41.542379 3565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:41.543493 3550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:39:41.543619 3550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "9553b930d3874fb5a9255cbc3742f7bd"
format_stamp: "Formatted at 2026-09-24 10:39:41 on dist-test-slave-j38h"
server_key: "84b367f36498a0c657b196e57abe8fa4"
server_key_iv: "4704105d480999aad6438f55df41ac4f"
server_key_version: "encryptionkey@0"
I20260924 10:39:41.543720 3550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:41.564807 3550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:41.567507 3550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:41.567718 3550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:41.569144 3550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:41.569218 3550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:41.569286 3550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:41.569346 3550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:39:41 dist-test-slave-j38h krb5kdc[3153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246381, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:41.580103 3550 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:45927
I20260924 10:39:41.580173 3680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:45927 every 8 connection(s)
I20260924 10:39:41.581207 3550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:39:41.583104 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.569998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:38355 (local address 127.2.8.62:42661)
0924 10:39:41.570123 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.570126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.571110 (+ 984us) server_negotiation.cc:407] Connection header received
0924 10:39:41.572127 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.572131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.572187 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.572261 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:41.573928 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.574815 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.576118 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.576515 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.576647 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.579404 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:41.579410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:41.579416 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:41.579450 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:41.581190 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:41.581743 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:41.581747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:41.581749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:41.581833 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:41.582196 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:41.582199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:41.582200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:41.582375 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:41.582456 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.582895 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.582990 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:39:41.583861 3574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.570520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:42661 (local address 127.2.8.3:38355)
0924 10:39:41.570962 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:41.571003 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:41.571880 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:41.572753 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:41.572759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:41.573211 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:41.573756 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.573774 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.574965 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.574968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.575985 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.575994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.576665 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.576724 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.576940 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.577586 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:41.577602 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:41.579238 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:41.581298 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:41.581304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:41.581318 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:41.581599 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:41.581936 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:41.581939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:41.581941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:41.582092 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:41.582472 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:41.582476 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:41.582683 (+ 207us) client_negotiation.cc:769] Sending connection context
0924 10:39:41.582832 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 10:39:41.583056 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":125,"threads_started":1}
I20260924 10:39:41.585203 3681 heartbeater.cc:347] Connected to a master server at 127.2.8.62:42661
I20260924 10:39:41.585357 3681 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:41.585826 3681 heartbeater.cc:511] Master 127.2.8.62:42661 requested a full tablet report, sending...
I20260924 10:39:41.586834 3198 ts_manager.cc:194] Registered new tserver with Master: 9553b930d3874fb5a9255cbc3742f7bd (127.2.8.3:45927)
I20260924 10:39:41.587612 3198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:38355
I20260924 10:39:41.588781 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 3550
I20260924 10:39:41.588882 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:39:41.589190 2080 external_mini_cluster.cc:1428] Setting key ae994dd94eb28aec7d9bbccf5094a58e
I20260924 10:39:41.591213 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:39:41.594368 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.587218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:42661 (local address 127.2.8.3:50305)
0924 10:39:41.587501 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:41.587514 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:41.587604 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:41.587874 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:41.587878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:41.588057 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:41.588168 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.588176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.588870 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.588873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.589553 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.589563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.590103 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.590119 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.590214 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.590742 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:41.590753 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:41.591160 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:41.593023 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:41.593027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:41.593029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:41.593215 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:41.593500 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:41.593502 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:41.593503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:41.593573 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:41.594006 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:41.594008 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:41.594075 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 10:39:41.594152 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 10:39:41.594246 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":93,"threads_started":1}
I20260924 10:39:41.594419 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.587278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:50305 (local address 127.2.8.62:42661)
0924 10:39:41.587413 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.587416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.587565 (+ 149us) server_negotiation.cc:407] Connection header received
0924 10:39:41.587703 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.587706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.587756 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.587839 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:41.588264 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.588738 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.589670 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.589991 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.590086 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.591299 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:41.591304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:41.591308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:41.591333 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:41.592856 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:41.593308 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:41.593312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:41.593316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:41.593406 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:41.593657 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:41.593663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:41.593665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:41.593904 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:41.594007 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.594214 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.594295 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 10:39:41 dist-test-slave-j38h krb5kdc[3153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246381, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:39:41 dist-test-slave-j38h krb5kdc[3153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246381, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:41.616243 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.608605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32818 (local address 127.2.8.62:42661)
0924 10:39:41.608733 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.608736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.608748 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:39:41.608781 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.608783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.608825 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.608891 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:41.609580 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.610084 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.610888 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.611095 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.611188 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.613430 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:41.613433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:41.613435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:41.613455 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:41.614718 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:41.615201 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:41.615203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:41.615204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:41.615270 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:41.615537 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:41.615539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:41.615540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:41.615736 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:41.615834 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.616045 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.616130 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:39:41.619091 3198 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:39:41.622143 3198 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:39:41.636808 3700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.630534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:45927 (local address 127.0.0.1:34748)
0924 10:39:41.631082 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:41.631130 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:41.631264 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:41.632538 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:41.632541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:41.632553 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:41.632732 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.632739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.633745 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.633751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.635205 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.635215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.636234 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.636238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.636353 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.636386 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 10:39:41.636512 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 10:39:41.636651 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":137,"thread_start_us":70,"threads_started":1}
I20260924 10:39:41.636853 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.630756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:35811 (local address 127.0.0.1:45368)
0924 10:39:41.631205 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:41.631216 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:41.631283 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:41.632445 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:41.632454 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:41.632483 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:41.632760 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.632768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.634193 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.634197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.635474 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.635482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.636499 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.636505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.636610 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.636629 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 10:39:41.636684 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 10:39:41.636737 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":209,"thread_start_us":99,"threads_started":1}
I20260924 10:39:41.636843 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.630700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34748 (local address 127.2.8.3:45927)
0924 10:39:41.632036 (+ 1336us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.632044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.632062 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:39:41.632119 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.632123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.632394 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.632700 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:41.632829 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.633637 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.635333 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.636098 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.636379 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.636556 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.636608 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.636703 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1255,"thread_start_us":110,"threads_started":1}
I20260924 10:39:41.636938 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.631640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.2.8.1:35811)
0924 10:39:41.631978 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.631991 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.632010 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:39:41.632069 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.632074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.632282 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.632443 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:41.632897 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.634074 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.635601 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.636368 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.636491 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.636663 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.636708 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.636783 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":70,"threads_started":1}
I20260924 10:39:41.637336 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.631307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:33291 (local address 127.0.0.1:56458)
0924 10:39:41.631536 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:41.631551 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:41.631630 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:41.632609 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:41.632612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:41.632637 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:41.632770 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.632775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.634234 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.634237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.635399 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.635407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.636179 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.636183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.636290 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.636975 (+ 685us) client_negotiation.cc:769] Sending connection context
0924 10:39:41.637099 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 10:39:41.637167 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":143,"thread_start_us":107,"threads_started":1}
I20260924 10:39:41.637738 3702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.631558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56458 (local address 127.2.8.2:33291)
0924 10:39:41.631955 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.631966 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.631990 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:39:41.632069 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.632074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.632289 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.632443 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:41.632897 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.634081 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.635518 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.636061 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.636181 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.636348 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.637493 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.637617 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":99,"threads_started":1}
I20260924 10:39:41.639972 3615 tablet_service.cc:1467] Processing CreateTablet for tablet dd296b59e52d4f5094f0028c8b6ee056 (DEFAULT_TABLE table=test-table [id=7f3750f6bf014d0391f96350fa64a353]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:41.640024 3335 tablet_service.cc:1467] Processing CreateTablet for tablet dd296b59e52d4f5094f0028c8b6ee056 (DEFAULT_TABLE table=test-table [id=7f3750f6bf014d0391f96350fa64a353]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:41.640439 3475 tablet_service.cc:1467] Processing CreateTablet for tablet dd296b59e52d4f5094f0028c8b6ee056 (DEFAULT_TABLE table=test-table [id=7f3750f6bf014d0391f96350fa64a353]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:41.640966 3615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd296b59e52d4f5094f0028c8b6ee056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:41.641024 3335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd296b59e52d4f5094f0028c8b6ee056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:41.641276 3475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd296b59e52d4f5094f0028c8b6ee056. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:41.647066 3704 tablet_bootstrap.cc:492] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd: Bootstrap starting.
I20260924 10:39:41.649166 3705 tablet_bootstrap.cc:492] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7: Bootstrap starting.
I20260924 10:39:41.649621 3704 tablet_bootstrap.cc:654] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:41.650465 3704 log.cc:824] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:41.650920 3706 tablet_bootstrap.cc:492] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Bootstrap starting.
I20260924 10:39:41.651171 3705 tablet_bootstrap.cc:654] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:41.651983 3705 log.cc:824] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:41.652427 3704 tablet_bootstrap.cc:492] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd: No bootstrap required, opened a new log
I20260924 10:39:41.652622 3704 ts_tablet_manager.cc:1397] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:39:41.653008 3706 tablet_bootstrap.cc:654] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:41.653810 3706 log.cc:824] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:41.654255 3705 tablet_bootstrap.cc:492] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7: No bootstrap required, opened a new log
I20260924 10:39:41.654474 3705 ts_tablet_manager.cc:1397] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:39:41.655565 3706 tablet_bootstrap.cc:492] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: No bootstrap required, opened a new log
I20260924 10:39:41.655743 3706 ts_tablet_manager.cc:1397] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260924 10:39:41.656072 3704 raft_consensus.cc:348] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } }
I20260924 10:39:41.656301 3704 raft_consensus.cc:374] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:41.656359 3704 raft_consensus.cc:729] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553b930d3874fb5a9255cbc3742f7bd, State: Initialized, Role: FOLLOWER
I20260924 10:39:41.656806 3704 consensus_queue.cc:260] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } }
I20260924 10:39:41.657416 3681 heartbeater.cc:503] Master 127.2.8.62:42661 was elected leader, sending a full tablet report...
I20260924 10:39:41.657781 3704 ts_tablet_manager.cc:1428] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:39:41.658255 3705 raft_consensus.cc:348] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } }
I20260924 10:39:41.658504 3705 raft_consensus.cc:374] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:41.658550 3705 raft_consensus.cc:729] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06de1d796776422abb72a1cd78b2bfc7, State: Initialized, Role: FOLLOWER
I20260924 10:39:41.659161 3705 consensus_queue.cc:260] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } }
I20260924 10:39:41.659569 3706 raft_consensus.cc:348] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } }
I20260924 10:39:41.659813 3706 raft_consensus.cc:374] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:41.659874 3706 raft_consensus.cc:729] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb8d14c65eb4fd8aa0df98b3ed9034d, State: Initialized, Role: FOLLOWER
I20260924 10:39:41.660283 3706 consensus_queue.cc:260] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } }
I20260924 10:39:41.660393 3705 ts_tablet_manager.cc:1428] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:39:41.660722 3541 heartbeater.cc:503] Master 127.2.8.62:42661 was elected leader, sending a full tablet report...
I20260924 10:39:41.660943 3401 heartbeater.cc:503] Master 127.2.8.62:42661 was elected leader, sending a full tablet report...
I20260924 10:39:41.661221 3706 ts_tablet_manager.cc:1428] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260924 10:39:41.691640 3712 raft_consensus.cc:482] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:39:41.691829 3712 raft_consensus.cc:504] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } }
I20260924 10:39:41.692938 3712 leader_election.cc:288] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9553b930d3874fb5a9255cbc3742f7bd (127.2.8.3:45927), 06de1d796776422abb72a1cd78b2bfc7 (127.2.8.2:33291)
I20260924 10:39:41.698266 3714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.693242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:45927 (local address 127.2.8.1:48025)
0924 10:39:41.693596 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:41.693612 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:41.693736 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:41.694039 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:41.694042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:41.694086 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:41.694294 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.694302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.695342 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.695346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.696866 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.696872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.697785 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.697790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.697884 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.697930 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 10:39:41.698008 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 10:39:41.698095 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":177,"spinlock_wait_cycles":9728,"thread_start_us":108,"threads_started":1}
I20260924 10:39:41.698263 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.693466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:48025 (local address 127.2.8.3:45927)
0924 10:39:41.693576 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.693580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.693694 (+ 114us) server_negotiation.cc:407] Connection header received
0924 10:39:41.693842 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.693845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.693897 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.693982 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:41.694420 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.695219 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.697012 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.697635 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.697738 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.697801 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.698047 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.698134 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 10:39:41.698463 3702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.693436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:42661 (local address 127.2.8.2:33291)
0924 10:39:41.693575 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.693580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.693734 (+ 154us) server_negotiation.cc:407] Connection header received
0924 10:39:41.693841 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.693845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.693897 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.693982 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:41.694421 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.695219 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.697015 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.697635 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.697739 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.697804 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.698201 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.698334 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:39:41.698877 3713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.693250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:33291 (local address 127.2.8.1:42661)
0924 10:39:41.693646 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:41.693659 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:41.693736 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:41.694039 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:41.694042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:41.694085 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:41.694304 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.694309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.695342 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.695346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.696861 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.696872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.697882 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.697885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.698051 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.698081 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:39:41.698651 (+ 570us) client_negotiation.cc:240] Negotiation successful
0924 10:39:41.698748 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":247,"spinlock_wait_cycles":64384,"thread_start_us":115,"threads_started":1}
I20260924 10:39:41.698990 3635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd296b59e52d4f5094f0028c8b6ee056" candidate_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553b930d3874fb5a9255cbc3742f7bd" is_pre_election: true
I20260924 10:39:41.699337 3635 raft_consensus.cc:2454] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb8d14c65eb4fd8aa0df98b3ed9034d in term 0.
I20260924 10:39:41.699595 3495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd296b59e52d4f5094f0028c8b6ee056" candidate_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06de1d796776422abb72a1cd78b2bfc7" is_pre_election: true
I20260924 10:39:41.699875 3287 leader_election.cc:302] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb8d14c65eb4fd8aa0df98b3ed9034d, 9553b930d3874fb5a9255cbc3742f7bd; no voters:
I20260924 10:39:41.699900 3495 raft_consensus.cc:2454] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb8d14c65eb4fd8aa0df98b3ed9034d in term 0.
I20260924 10:39:41.700178 3712 raft_consensus.cc:2781] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:39:41.700289 3712 raft_consensus.cc:482] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:39:41.700397 3712 raft_consensus.cc:3037] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:41.701526 3712 raft_consensus.cc:504] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } }
I20260924 10:39:41.701853 3712 leader_election.cc:288] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [CANDIDATE]: Term 1 election: Requested vote from peers 9553b930d3874fb5a9255cbc3742f7bd (127.2.8.3:45927), 06de1d796776422abb72a1cd78b2bfc7 (127.2.8.2:33291)
I20260924 10:39:41.702153 3495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd296b59e52d4f5094f0028c8b6ee056" candidate_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06de1d796776422abb72a1cd78b2bfc7"
I20260924 10:39:41.702173 3635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd296b59e52d4f5094f0028c8b6ee056" candidate_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553b930d3874fb5a9255cbc3742f7bd"
I20260924 10:39:41.702261 3495 raft_consensus.cc:3037] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:41.702344 3635 raft_consensus.cc:3037] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:41.703377 3495 raft_consensus.cc:2454] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb8d14c65eb4fd8aa0df98b3ed9034d in term 1.
I20260924 10:39:41.703460 3635 raft_consensus.cc:2454] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb8d14c65eb4fd8aa0df98b3ed9034d in term 1.
I20260924 10:39:41.703713 3286 leader_election.cc:302] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06de1d796776422abb72a1cd78b2bfc7, 2cb8d14c65eb4fd8aa0df98b3ed9034d; no voters:
I20260924 10:39:41.703914 3712 raft_consensus.cc:2781] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:41.704105 3712 raft_consensus.cc:686] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 1 LEADER]: Becoming Leader. State: Replica: 2cb8d14c65eb4fd8aa0df98b3ed9034d, State: Running, Role: LEADER
I20260924 10:39:41.704429 3712 consensus_queue.cc:237] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } }
I20260924 10:39:41.707793 3196 catalog_manager.cc:5495] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cb8d14c65eb4fd8aa0df98b3ed9034d (127.2.8.1). New cstate: current_term: 1 leader_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } health_report { overall_health: UNKNOWN } } }
I20260924 10:39:41.726954 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.723549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45380 (local address 127.2.8.1:35811)
0924 10:39:41.723678 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.723682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.723695 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:39:41.723760 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.723762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.723839 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.723937 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:41.724168 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.724675 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.725555 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.725760 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.725839 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.726120 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:41.726521 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:41.726606 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.726698 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.726832 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:39:41.733173 3635 raft_consensus.cc:1261] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd [term 1 FOLLOWER]: Refusing update from remote peer 2cb8d14c65eb4fd8aa0df98b3ed9034d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:41.733173 3495 raft_consensus.cc:1261] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7 [term 1 FOLLOWER]: Refusing update from remote peer 2cb8d14c65eb4fd8aa0df98b3ed9034d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:41.733907 3715 consensus_queue.cc:1035] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:41.734127 3712 consensus_queue.cc:1035] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [LEADER]: Connected to new peer: Peer: permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:41.753135 3198 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:39:41.753340 3198 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:39:41.755779 3198 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443: Sending DeleteTablet for 3 replicas of tablet dd296b59e52d4f5094f0028c8b6ee056
I20260924 10:39:41.756538 3615 tablet_service.cc:1514] Processing DeleteTablet for tablet dd296b59e52d4f5094f0028c8b6ee056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:41 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:34748
I20260924 10:39:41.756578 3475 tablet_service.cc:1514] Processing DeleteTablet for tablet dd296b59e52d4f5094f0028c8b6ee056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:41 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:56458
I20260924 10:39:41.757081 3728 tablet_replica.cc:330] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7: stopping tablet replica
I20260924 10:39:41.757228 3335 tablet_service.cc:1514] Processing DeleteTablet for tablet dd296b59e52d4f5094f0028c8b6ee056 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:41 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:45368
I20260924 10:39:41.757534 3728 raft_consensus.cc:2229] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:39:41.757675 3730 tablet_replica.cc:330] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: stopping tablet replica
I20260924 10:39:41.757957 3729 tablet_replica.cc:330] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd: stopping tablet replica
I20260924 10:39:41.757959 3730 raft_consensus.cc:2229] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:39:41.757942 3198 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 10:39:41.758201 3729 raft_consensus.cc:2229] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:39:41.758325 3730 raft_consensus.cc:2258] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 10:39:41.758332 3198 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:39:41.758463 3729 raft_consensus.cc:2258] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:41.759874 3729 ts_tablet_manager.cc:1905] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:41.759874 3730 ts_tablet_manager.cc:1905] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:41.762578 3730 ts_tablet_manager.cc:1918] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:39:41.762670 3730 log.cc:1196] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/wal/wals/dd296b59e52d4f5094f0028c8b6ee056
I20260924 10:39:41.762982 3730 ts_tablet_manager.cc:1939] T dd296b59e52d4f5094f0028c8b6ee056 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Deleting consensus metadata
I20260924 10:39:41.763671 3335 tablet_service.cc:1467] Processing CreateTablet for tablet 07e9ed5056fb415a8dac97d5c04add05 (DEFAULT_TABLE table=test-table [id=a8fc20d928f34cd6bb08eed754a5e2c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:41.763914 3335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e9ed5056fb415a8dac97d5c04add05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:41.763865 3475 tablet_service.cc:1467] Processing CreateTablet for tablet 07e9ed5056fb415a8dac97d5c04add05 (DEFAULT_TABLE table=test-table [id=a8fc20d928f34cd6bb08eed754a5e2c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:41.764132 3475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e9ed5056fb415a8dac97d5c04add05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:41.764140 3182 catalog_manager.cc:4841] TS 2cb8d14c65eb4fd8aa0df98b3ed9034d (127.2.8.1:35811): tablet dd296b59e52d4f5094f0028c8b6ee056 (table test-table [id=7f3750f6bf014d0391f96350fa64a353]) successfully deleted
I20260924 10:39:41.765451 3729 ts_tablet_manager.cc:1918] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:39:41.765563 3729 log.cc:1196] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/dd296b59e52d4f5094f0028c8b6ee056
I20260924 10:39:41.765844 3729 ts_tablet_manager.cc:1939] T dd296b59e52d4f5094f0028c8b6ee056 P 9553b930d3874fb5a9255cbc3742f7bd: Deleting consensus metadata
I20260924 10:39:41.766036 3706 tablet_bootstrap.cc:492] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Bootstrap starting.
I20260924 10:39:41.766505 3705 tablet_bootstrap.cc:492] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7: Bootstrap starting.
I20260924 10:39:41.766937 3706 tablet_bootstrap.cc:654] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:41.767431 3705 tablet_bootstrap.cc:654] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:41.768014 3183 catalog_manager.cc:4841] TS 9553b930d3874fb5a9255cbc3742f7bd (127.2.8.3:45927): tablet dd296b59e52d4f5094f0028c8b6ee056 (table test-table [id=7f3750f6bf014d0391f96350fa64a353]) successfully deleted
I20260924 10:39:41.768112 3706 tablet_bootstrap.cc:492] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: No bootstrap required, opened a new log
I20260924 10:39:41.768204 3706 ts_tablet_manager.cc:1397] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:41.768532 3615 tablet_service.cc:1467] Processing CreateTablet for tablet 07e9ed5056fb415a8dac97d5c04add05 (DEFAULT_TABLE table=test-table [id=a8fc20d928f34cd6bb08eed754a5e2c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:41.768647 3706 raft_consensus.cc:348] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.768747 3706 raft_consensus.cc:374] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:41.768769 3615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07e9ed5056fb415a8dac97d5c04add05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:41.768784 3706 raft_consensus.cc:729] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb8d14c65eb4fd8aa0df98b3ed9034d, State: Initialized, Role: FOLLOWER
I20260924 10:39:41.768911 3706 consensus_queue.cc:260] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.769209 3706 ts_tablet_manager.cc:1428] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:39:41.770923 3704 tablet_bootstrap.cc:492] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd: Bootstrap starting.
I20260924 10:39:41.771811 3704 tablet_bootstrap.cc:654] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:41.772562 3728 raft_consensus.cc:2258] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:41.772954 3704 tablet_bootstrap.cc:492] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd: No bootstrap required, opened a new log
I20260924 10:39:41.773062 3704 ts_tablet_manager.cc:1397] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:41.773479 3704 raft_consensus.cc:348] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.773577 3704 raft_consensus.cc:374] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:41.773610 3704 raft_consensus.cc:729] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553b930d3874fb5a9255cbc3742f7bd, State: Initialized, Role: FOLLOWER
I20260924 10:39:41.773722 3704 consensus_queue.cc:260] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.773972 3704 ts_tablet_manager.cc:1428] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:39:41.774343 3728 ts_tablet_manager.cc:1905] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:41.776597 3728 ts_tablet_manager.cc:1918] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:39:41.776669 3728 log.cc:1196] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/dd296b59e52d4f5094f0028c8b6ee056
I20260924 10:39:41.777035 3728 ts_tablet_manager.cc:1939] T dd296b59e52d4f5094f0028c8b6ee056 P 06de1d796776422abb72a1cd78b2bfc7: Deleting consensus metadata
I20260924 10:39:41.778172 3182 catalog_manager.cc:4841] TS 06de1d796776422abb72a1cd78b2bfc7 (127.2.8.2:33291): tablet dd296b59e52d4f5094f0028c8b6ee056 (table test-table [id=7f3750f6bf014d0391f96350fa64a353]) successfully deleted
I20260924 10:39:41.778612 3705 tablet_bootstrap.cc:492] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7: No bootstrap required, opened a new log
I20260924 10:39:41.778683 3705 ts_tablet_manager.cc:1397] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260924 10:39:41.779143 3705 raft_consensus.cc:348] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.779258 3705 raft_consensus.cc:374] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:41.779296 3705 raft_consensus.cc:729] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06de1d796776422abb72a1cd78b2bfc7, State: Initialized, Role: FOLLOWER
I20260924 10:39:41.779421 3705 consensus_queue.cc:260] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.779664 3705 ts_tablet_manager.cc:1428] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:39:41.806666 3711 raft_consensus.cc:482] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:39:41.806830 3711 raft_consensus.cc:504] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.808101 3711 leader_election.cc:288] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cb8d14c65eb4fd8aa0df98b3ed9034d (127.2.8.1:35811), 9553b930d3874fb5a9255cbc3742f7bd (127.2.8.3:45927)
I20260924 10:39:41.812004 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.808815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:46723 (local address 127.2.8.3:45927)
0924 10:39:41.808961 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.808966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.808979 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:39:41.809040 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.809043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.809087 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.809162 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:41.809552 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.810046 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.811074 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.811465 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.811582 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.811641 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.811818 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.811894 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:39:41.812008 3434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.808345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:45927 (local address 127.2.8.2:46723)
0924 10:39:41.808455 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:41.808472 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:41.808558 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:41.809278 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:41.809281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:41.809300 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:41.809430 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.809434 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.810155 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.810161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.810947 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.810954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.811583 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.811586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.811683 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.811710 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 10:39:41.811792 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 10:39:41.811874 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":4,"spinlock_wait_cycles":1408}
I20260924 10:39:41.812474 3635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e9ed5056fb415a8dac97d5c04add05" candidate_uuid: "06de1d796776422abb72a1cd78b2bfc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553b930d3874fb5a9255cbc3742f7bd" is_pre_election: true
I20260924 10:39:41.812626 3635 raft_consensus.cc:2454] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06de1d796776422abb72a1cd78b2bfc7 in term 0.
I20260924 10:39:41.813009 3427 leader_election.cc:302] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06de1d796776422abb72a1cd78b2bfc7, 9553b930d3874fb5a9255cbc3742f7bd; no voters:
I20260924 10:39:41.813129 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.808453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:39245 (local address 127.2.8.1:35811)
0924 10:39:41.808567 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.808571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.808835 (+ 264us) server_negotiation.cc:407] Connection header received
0924 10:39:41.809024 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.809028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.809119 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.809188 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:41.809563 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.810337 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.811864 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.812462 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.812533 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.812590 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.812915 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.813009 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:39:41.813177 3734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.808345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:35811 (local address 127.2.8.2:39245)
0924 10:39:41.808714 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:41.808728 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:41.808841 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:41.809253 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:41.809256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:41.809275 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:41.809435 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.809441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.810476 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.810479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.811720 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.811729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.812695 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.812698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.812806 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.812831 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:39:41.812896 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 10:39:41.812960 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":101,"spinlock_wait_cycles":11008,"thread_start_us":104,"threads_started":1}
I20260924 10:39:41.813283 3711 raft_consensus.cc:2781] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:39:41.813330 3711 raft_consensus.cc:482] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:39:41.813366 3711 raft_consensus.cc:3037] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:41.813674 3355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e9ed5056fb415a8dac97d5c04add05" candidate_uuid: "06de1d796776422abb72a1cd78b2bfc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" is_pre_election: true
I20260924 10:39:41.813918 3355 raft_consensus.cc:2454] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06de1d796776422abb72a1cd78b2bfc7 in term 0.
I20260924 10:39:41.814554 3711 raft_consensus.cc:504] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.814913 3711 leader_election.cc:288] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [CANDIDATE]: Term 1 election: Requested vote from peers 2cb8d14c65eb4fd8aa0df98b3ed9034d (127.2.8.1:35811), 9553b930d3874fb5a9255cbc3742f7bd (127.2.8.3:45927)
I20260924 10:39:41.815189 3355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e9ed5056fb415a8dac97d5c04add05" candidate_uuid: "06de1d796776422abb72a1cd78b2bfc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d"
I20260924 10:39:41.815296 3355 raft_consensus.cc:3037] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:41.815372 3635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07e9ed5056fb415a8dac97d5c04add05" candidate_uuid: "06de1d796776422abb72a1cd78b2bfc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553b930d3874fb5a9255cbc3742f7bd"
I20260924 10:39:41.815503 3635 raft_consensus.cc:3037] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:41.816365 3635 raft_consensus.cc:2454] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06de1d796776422abb72a1cd78b2bfc7 in term 1.
I20260924 10:39:41.816429 3355 raft_consensus.cc:2454] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06de1d796776422abb72a1cd78b2bfc7 in term 1.
I20260924 10:39:41.816671 3427 leader_election.cc:302] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06de1d796776422abb72a1cd78b2bfc7, 9553b930d3874fb5a9255cbc3742f7bd; no voters:
I20260924 10:39:41.816839 3711 raft_consensus.cc:2781] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:41.817023 3711 raft_consensus.cc:686] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [term 1 LEADER]: Becoming Leader. State: Replica: 06de1d796776422abb72a1cd78b2bfc7, State: Running, Role: LEADER
I20260924 10:39:41.817257 3711 consensus_queue.cc:237] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.819778 3197 catalog_manager.cc:5495] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06de1d796776422abb72a1cd78b2bfc7 (127.2.8.2). New cstate: current_term: 1 leader_uuid: "06de1d796776422abb72a1cd78b2bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } health_report { overall_health: UNKNOWN } } }
W20260924 10:39:41.832231 3682 tablet.cc:2369] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:39:41 dist-test-slave-j38h krb5kdc[3153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246380, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:41.862376 3742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.850990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:42661 (local address 127.0.0.1:32828)
0924 10:39:41.851563 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:41.851580 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:41.851721 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:41.852111 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:41.852115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:41.852630 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:41.852751 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.852759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.853968 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.853972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.854888 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.854911 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.855649 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.855671 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.855750 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.856503 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:41.856518 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:41.858256 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:41.860573 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:41.860580 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:41.860586 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:41.860892 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:41.861269 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:41.861272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:41.861274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:41.861366 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:41.861794 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:41.861801 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:41.861922 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 10:39:41.862035 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 10:39:41.862186 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":113,"threads_started":1}
I20260924 10:39:41.862480 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.851029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.2.8.62:42661)
0924 10:39:41.851165 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.851172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.851655 (+ 483us) server_negotiation.cc:407] Connection header received
0924 10:39:41.851830 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.851836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.851909 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.852007 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:41.852905 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.853819 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.855104 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.855436 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.855514 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.858397 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:41.858401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:41.858403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:41.858428 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:41.860455 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:41.861035 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:41.861041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:41.861045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:41.861152 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:41.861461 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:41.861464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:41.861466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:41.861680 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:41.861784 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.862110 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.862267 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":48}
W20260924 10:39:41.863623 3542 tablet.cc:2369] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:39:41.867110 3197 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:32828:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 10:39:41.867938 3197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:39:41.873071 3475 tablet_service.cc:1467] Processing CreateTablet for tablet 5332a003fafe43f39b557f8bf23b0e07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6f339349a46545f6b25827f854ca6d38]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:39:41.873090 3615 tablet_service.cc:1467] Processing CreateTablet for tablet 5332a003fafe43f39b557f8bf23b0e07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6f339349a46545f6b25827f854ca6d38]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:39:41.873160 3335 tablet_service.cc:1467] Processing CreateTablet for tablet 5332a003fafe43f39b557f8bf23b0e07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6f339349a46545f6b25827f854ca6d38]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:39:41.873474 3615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5332a003fafe43f39b557f8bf23b0e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:41.873474 3335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5332a003fafe43f39b557f8bf23b0e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:41.873477 3475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5332a003fafe43f39b557f8bf23b0e07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:41.876219 3706 tablet_bootstrap.cc:492] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Bootstrap starting.
I20260924 10:39:41.876436 3705 tablet_bootstrap.cc:492] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7: Bootstrap starting.
I20260924 10:39:41.876502 3704 tablet_bootstrap.cc:492] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd: Bootstrap starting.
I20260924 10:39:41.877288 3706 tablet_bootstrap.cc:654] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:41.877414 3705 tablet_bootstrap.cc:654] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:41.877595 3704 tablet_bootstrap.cc:654] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:41.878854 3704 tablet_bootstrap.cc:492] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd: No bootstrap required, opened a new log
I20260924 10:39:41.878947 3704 ts_tablet_manager.cc:1397] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 10:39:41.879451 3706 tablet_bootstrap.cc:492] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: No bootstrap required, opened a new log
I20260924 10:39:41.879529 3706 ts_tablet_manager.cc:1397] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:39:41.879451 3704 raft_consensus.cc:348] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.879568 3704 raft_consensus.cc:374] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:41.879601 3704 raft_consensus.cc:729] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9553b930d3874fb5a9255cbc3742f7bd, State: Initialized, Role: FOLLOWER
I20260924 10:39:41.879738 3704 consensus_queue.cc:260] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.880031 3704 ts_tablet_manager.cc:1428] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:39:41.880004 3706 raft_consensus.cc:348] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.880110 3706 raft_consensus.cc:374] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:41.880051 3710 tablet_replica.cc:439] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } }
I20260924 10:39:41.880147 3706 raft_consensus.cc:729] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb8d14c65eb4fd8aa0df98b3ed9034d, State: Initialized, Role: FOLLOWER
I20260924 10:39:41.880213 3710 tablet_replica.cc:442] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:39:41.880287 3706 consensus_queue.cc:260] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.880580 3706 ts_tablet_manager.cc:1428] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:39:41.880846 3715 tablet_replica.cc:439] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } }
I20260924 10:39:41.880955 3715 tablet_replica.cc:442] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:39:41.881539 3705 tablet_bootstrap.cc:492] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7: No bootstrap required, opened a new log
I20260924 10:39:41.881633 3705 ts_tablet_manager.cc:1397] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 10:39:41.882076 3705 raft_consensus.cc:348] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.882200 3705 raft_consensus.cc:374] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:41.882234 3705 raft_consensus.cc:729] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06de1d796776422abb72a1cd78b2bfc7, State: Initialized, Role: FOLLOWER
I20260924 10:39:41.882356 3705 consensus_queue.cc:260] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.882642 3705 ts_tablet_manager.cc:1428] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:39:41.884832 3711 tablet_replica.cc:439] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } }
I20260924 10:39:41.884963 3711 tablet_replica.cc:442] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 10:39:41.900087 3402 tablet.cc:2369] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:39:41.961959 3711 raft_consensus.cc:482] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:39:41.962083 3711 raft_consensus.cc:504] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.962476 3711 leader_election.cc:288] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cb8d14c65eb4fd8aa0df98b3ed9034d (127.2.8.1:35811), 9553b930d3874fb5a9255cbc3742f7bd (127.2.8.3:45927)
I20260924 10:39:41.962881 3355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332a003fafe43f39b557f8bf23b0e07" candidate_uuid: "06de1d796776422abb72a1cd78b2bfc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" is_pre_election: true
I20260924 10:39:41.962957 3635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332a003fafe43f39b557f8bf23b0e07" candidate_uuid: "06de1d796776422abb72a1cd78b2bfc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553b930d3874fb5a9255cbc3742f7bd" is_pre_election: true
I20260924 10:39:41.963042 3355 raft_consensus.cc:2454] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06de1d796776422abb72a1cd78b2bfc7 in term 0.
I20260924 10:39:41.963079 3635 raft_consensus.cc:2454] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06de1d796776422abb72a1cd78b2bfc7 in term 0.
I20260924 10:39:41.963330 3426 leader_election.cc:302] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06de1d796776422abb72a1cd78b2bfc7, 2cb8d14c65eb4fd8aa0df98b3ed9034d; no voters:
I20260924 10:39:41.963493 3711 raft_consensus.cc:2781] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:39:41.963555 3711 raft_consensus.cc:482] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:39:41.963604 3711 raft_consensus.cc:3037] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:41.964586 3711 raft_consensus.cc:504] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.964951 3711 leader_election.cc:288] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [CANDIDATE]: Term 1 election: Requested vote from peers 2cb8d14c65eb4fd8aa0df98b3ed9034d (127.2.8.1:35811), 9553b930d3874fb5a9255cbc3742f7bd (127.2.8.3:45927)
I20260924 10:39:41.965310 3355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332a003fafe43f39b557f8bf23b0e07" candidate_uuid: "06de1d796776422abb72a1cd78b2bfc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d"
I20260924 10:39:41.965363 3635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5332a003fafe43f39b557f8bf23b0e07" candidate_uuid: "06de1d796776422abb72a1cd78b2bfc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9553b930d3874fb5a9255cbc3742f7bd"
I20260924 10:39:41.965440 3355 raft_consensus.cc:3037] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:41.965483 3635 raft_consensus.cc:3037] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:41.966550 3355 raft_consensus.cc:2454] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06de1d796776422abb72a1cd78b2bfc7 in term 1.
I20260924 10:39:41.966550 3635 raft_consensus.cc:2454] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06de1d796776422abb72a1cd78b2bfc7 in term 1.
I20260924 10:39:41.966838 3426 leader_election.cc:302] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06de1d796776422abb72a1cd78b2bfc7, 2cb8d14c65eb4fd8aa0df98b3ed9034d; no voters:
I20260924 10:39:41.967048 3711 raft_consensus.cc:2781] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:41.967137 3711 raft_consensus.cc:686] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [term 1 LEADER]: Becoming Leader. State: Replica: 06de1d796776422abb72a1cd78b2bfc7, State: Running, Role: LEADER
I20260924 10:39:41.967278 3711 consensus_queue.cc:237] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } }
I20260924 10:39:41.967923 3735 tablet_replica.cc:439] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7: TxnStatusTablet state changed. Reason: New leader 06de1d796776422abb72a1cd78b2bfc7. Latest consensus state: current_term: 1 leader_uuid: "06de1d796776422abb72a1cd78b2bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } }
I20260924 10:39:41.968027 3735 tablet_replica.cc:442] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:39:41.968251 3747 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 10:39:41.968470 3747 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 10:39:41.969091 3197 catalog_manager.cc:5495] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06de1d796776422abb72a1cd78b2bfc7 (127.2.8.2). New cstate: current_term: 1 leader_uuid: "06de1d796776422abb72a1cd78b2bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } health_report { overall_health: UNKNOWN } } }
I20260924 10:39:41.988453 3742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.984740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:33291 (local address 127.0.0.1:56466)
0924 10:39:41.984873 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:41.984897 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:41.985025 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:41.985338 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:41.985341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:41.985349 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:39:41.985454 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.985459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.986274 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.986279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.986906 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:41.986914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.987348 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.987363 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:41.987451 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.987502 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:39:41.988164 (+ 662us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:39:41.988178 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 10:39:41.988245 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 10:39:41.988330 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 10:39:41.988474 3702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:41.984882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.2.8.2:33291)
0924 10:39:41.985029 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:41.985034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:41.985048 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:39:41.985172 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:41.985174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:41.985227 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:41.985327 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:41.985576 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.986172 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.987025 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:41.987254 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:41.987360 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:41.987643 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:41.988059 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:41.988179 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:41.988255 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 10:39:41.988305 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 10:39:41.991297 3635 raft_consensus.cc:1261] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd [term 1 FOLLOWER]: Refusing update from remote peer 06de1d796776422abb72a1cd78b2bfc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:41.991506 3355 raft_consensus.cc:1261] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 1 FOLLOWER]: Refusing update from remote peer 06de1d796776422abb72a1cd78b2bfc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:41.991770 3735 consensus_queue.cc:1035] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:41.991987 3711 consensus_queue.cc:1035] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:41.993590 3715 tablet_replica.cc:439] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: TxnStatusTablet state changed. Reason: New leader 06de1d796776422abb72a1cd78b2bfc7. Latest consensus state: current_term: 1 leader_uuid: "06de1d796776422abb72a1cd78b2bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } }
I20260924 10:39:41.993613 3710 tablet_replica.cc:439] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd: TxnStatusTablet state changed. Reason: New leader 06de1d796776422abb72a1cd78b2bfc7. Latest consensus state: current_term: 1 leader_uuid: "06de1d796776422abb72a1cd78b2bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } }
I20260924 10:39:41.993701 3715 tablet_replica.cc:442] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:39:41.993711 3710 tablet_replica.cc:442] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:39:41.995282 3711 tablet_replica.cc:439] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06de1d796776422abb72a1cd78b2bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } }
I20260924 10:39:41.995383 3711 tablet_replica.cc:442] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:39:41.995563 3720 mvcc.cc:205] Tried to move back new op lower bound from 7332849180631445504 to 7332849180539342848. Current Snapshot: MvccSnapshot[applied={T|T < 7332849180631445504 or (T in {7332849180631445504})}]
I20260924 10:39:41.996611 3715 tablet_replica.cc:439] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06de1d796776422abb72a1cd78b2bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } }
I20260924 10:39:41.996709 3715 tablet_replica.cc:442] T 5332a003fafe43f39b557f8bf23b0e07 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:39:41.996680 3718 mvcc.cc:205] Tried to move back new op lower bound from 7332849180631445504 to 7332849180539342848. Current Snapshot: MvccSnapshot[applied={T|T < 7332849180631445504}]
I20260924 10:39:41.998328 3735 tablet_replica.cc:439] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06de1d796776422abb72a1cd78b2bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } }
I20260924 10:39:41.998436 3735 tablet_replica.cc:442] T 5332a003fafe43f39b557f8bf23b0e07 P 06de1d796776422abb72a1cd78b2bfc7: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:39:41.998914 3710 tablet_replica.cc:439] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06de1d796776422abb72a1cd78b2bfc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 } } peers { permanent_uuid: "06de1d796776422abb72a1cd78b2bfc7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33291 } } peers { permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 } } }
I20260924 10:39:41.999018 3710 tablet_replica.cc:442] T 5332a003fafe43f39b557f8bf23b0e07 P 9553b930d3874fb5a9255cbc3742f7bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:39:41.999557 3719 mvcc.cc:205] Tried to move back new op lower bound from 7332849180631445504 to 7332849180539342848. Current Snapshot: MvccSnapshot[applied={T|T < 7332849180631445504 or (T in {7332849180631445504})}]
I20260924 10:39:42.003924 3702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:42.000418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.2.8.2:33291)
0924 10:39:42.000748 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:42.000752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:42.000765 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:39:42.000806 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:42.000809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:42.000853 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:42.000930 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:42.001306 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:42.002079 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:42.002864 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:42.003214 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:42.003326 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:42.003530 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:42.003611 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:42.003683 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:42.003753 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 10:39:42.003800 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260924 10:39:42.014868 3702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:42.011169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:56515 (local address 127.2.8.2:33291)
0924 10:39:42.011291 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:42.011295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:42.011498 (+ 203us) server_negotiation.cc:407] Connection header received
0924 10:39:42.011579 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:42.011582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:42.011627 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:42.011702 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:42.011913 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:42.012662 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:42.013625 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:42.013994 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:42.014106 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:42.014451 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:42.014527 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:42.014599 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:42.014683 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 10:39:42.014727 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:39:42.014874 3754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:42.011074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:33291 (local address 127.2.8.2:56515)
0924 10:39:42.011420 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:42.011432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:42.011514 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:42.011716 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:42.011718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:42.011723 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:39:42.011821 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:42.011825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:42.012786 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:42.012789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:42.013506 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:42.013515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:42.014125 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:42.014142 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:42.014287 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:42.014328 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:39:42.014608 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:39:42.014612 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:39:42.014680 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 10:39:42.014743 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":97,"threads_started":1}
I20260924 10:39:42.021924 3635 raft_consensus.cc:1261] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd [term 1 FOLLOWER]: Refusing update from remote peer 06de1d796776422abb72a1cd78b2bfc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:42.022032 3355 raft_consensus.cc:1261] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 1 FOLLOWER]: Refusing update from remote peer 06de1d796776422abb72a1cd78b2bfc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:42.022315 3711 consensus_queue.cc:1035] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9553b930d3874fb5a9255cbc3742f7bd" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:42.022459 3735 consensus_queue.cc:1035] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb8d14c65eb4fd8aa0df98b3ed9034d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:42.063441 3198 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:39:42.063613 3198 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:39:42.065810 3198 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 89e89066249843bca8b9b843d4b0a443: Sending DeleteTablet for 3 replicas of tablet 07e9ed5056fb415a8dac97d5c04add05
I20260924 10:39:42.066412 3615 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e9ed5056fb415a8dac97d5c04add05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:42 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:34748
I20260924 10:39:42.066440 3335 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e9ed5056fb415a8dac97d5c04add05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:42 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:45368
I20260924 10:39:42.066591 3729 tablet_replica.cc:330] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd: stopping tablet replica
I20260924 10:39:42.066619 3730 tablet_replica.cc:330] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: stopping tablet replica
I20260924 10:39:42.066710 3729 raft_consensus.cc:2229] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:39:42.066733 3730 raft_consensus.cc:2229] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:39:42.066845 3729 raft_consensus.cc:2258] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:42.066919 3730 raft_consensus.cc:2258] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:42.067262 3474 tablet_service.cc:1514] Processing DeleteTablet for tablet 07e9ed5056fb415a8dac97d5c04add05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:42 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:56458
I20260924 10:39:42.067467 3728 tablet_replica.cc:330] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7: stopping tablet replica
I20260924 10:39:42.067564 3728 raft_consensus.cc:2229] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:39:42.067842 3728 raft_consensus.cc:2258] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:42.068636 3728 ts_tablet_manager.cc:1905] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:42.068636 3730 ts_tablet_manager.cc:1905] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:42.071336 3728 ts_tablet_manager.cc:1918] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:39:42.071336 3730 ts_tablet_manager.cc:1918] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:39:42.071403 3728 log.cc:1196] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/07e9ed5056fb415a8dac97d5c04add05
I20260924 10:39:42.071403 3730 log.cc:1196] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-0/wal/wals/07e9ed5056fb415a8dac97d5c04add05
I20260924 10:39:42.071655 3730 ts_tablet_manager.cc:1939] T 07e9ed5056fb415a8dac97d5c04add05 P 2cb8d14c65eb4fd8aa0df98b3ed9034d: Deleting consensus metadata
I20260924 10:39:42.071655 3728 ts_tablet_manager.cc:1939] T 07e9ed5056fb415a8dac97d5c04add05 P 06de1d796776422abb72a1cd78b2bfc7: Deleting consensus metadata
I20260924 10:39:42.072415 3729 ts_tablet_manager.cc:1905] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:42.072459 3182 catalog_manager.cc:4841] TS 2cb8d14c65eb4fd8aa0df98b3ed9034d (127.2.8.1:35811): tablet 07e9ed5056fb415a8dac97d5c04add05 (table test-table [id=a8fc20d928f34cd6bb08eed754a5e2c4]) successfully deleted
I20260924 10:39:42.075057 3729 ts_tablet_manager.cc:1918] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:39:42.075135 3729 log.cc:1196] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/07e9ed5056fb415a8dac97d5c04add05
I20260924 10:39:42.075393 3729 ts_tablet_manager.cc:1939] T 07e9ed5056fb415a8dac97d5c04add05 P 9553b930d3874fb5a9255cbc3742f7bd: Deleting consensus metadata
I20260924 10:39:42.076035 3183 catalog_manager.cc:4841] TS 9553b930d3874fb5a9255cbc3742f7bd (127.2.8.3:45927): tablet 07e9ed5056fb415a8dac97d5c04add05 (table test-table [id=a8fc20d928f34cd6bb08eed754a5e2c4]) successfully deleted
I20260924 10:39:42.076376 3182 catalog_manager.cc:4841] TS 06de1d796776422abb72a1cd78b2bfc7 (127.2.8.2:33291): tablet 07e9ed5056fb415a8dac97d5c04add05 (table test-table [id=a8fc20d928f34cd6bb08eed754a5e2c4]) successfully deleted
Sep 24 10:39:42 dist-test-slave-j38h krb5kdc[3153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246381, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.8.1@KRBTEST.COM
I20260924 10:39:42.083303 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:42.072349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.2.8.1:35811)
0924 10:39:42.072462 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:42.072466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:42.073054 (+ 588us) server_negotiation.cc:407] Connection header received
0924 10:39:42.073210 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:42.073213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:42.073263 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:42.073569 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:42.074308 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:42.074820 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:42.075803 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:42.076122 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:42.076218 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:42.079380 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:42.079394 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:42.079411 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:42.079462 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:42.081554 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:42.081985 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:42.081992 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:42.081994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:42.082084 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:42.082369 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:42.082374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:42.082376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:42.082627 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:42.082872 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:42.083023 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 10:39:42.083144 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 10:39:42.083871 3395 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45394
I20260924 10:39:42.093053 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:42.085617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.2.8.62:42661)
0924 10:39:42.085751 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:42.085756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:42.085927 (+ 171us) server_negotiation.cc:407] Connection header received
0924 10:39:42.086042 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:42.086044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:42.086087 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:42.086184 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:42.086819 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:42.087348 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:42.088137 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:42.088376 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:42.088486 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:42.089857 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:42.089861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:42.089863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:42.089889 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:42.091373 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:42.091790 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:42.091795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:42.091796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:42.091879 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:42.092149 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:42.092154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:42.092156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:42.092339 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:42.092477 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:42.092716 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 10:39:42.092835 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 10:39:42.093500 3198 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32832
I20260924 10:39:42.095093 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 3270
I20260924 10:39:42.102417 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 3410
I20260924 10:39:42.107941 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 3550
I20260924 10:39:42.113632 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 3166
2026-09-24T10:39:42Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2618 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:42 dist-test-slave-j38h krb5kdc[3767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:42 dist-test-slave-j38h krb5kdc[3767](info): set up 2 sockets
Sep 24 10:39:42 dist-test-slave-j38h krb5kdc[3767](info): commencing operation
krb5kdc: starting...
W20260924 10:39:43.200905 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:39:43 dist-test-slave-j38h krb5kdc[3767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:39:43.323942 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:43213
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:40333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:43213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:43.433988 3780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:43.434304 3780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:43.434404 3780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:43.438387 3780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:43.438486 3780 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 10:39:43.438517 3780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:43.438560 3780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:43.438594 3780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:43.438629 3780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:43.443866 3780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:40333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:43213
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:43213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:43.445032 3780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:43.446112 3780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:43.454437 3788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:43.456074 3786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:43.456573 3780 server_base.cc:1034] running on GCE node
W20260924 10:39:43.456645 3785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:43.457293 3780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:43.458439 3780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:43.459630 3780 hybrid_clock.cc:648] HybridClock initialized: now 1790246383459606 us; error 37 us; skew 500 ppm
Sep 24 10:39:43 dist-test-slave-j38h krb5kdc[3767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:43.462627 3780 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:39:43.463619 3780 webserver.cc:460] Webserver started at http://127.2.8.62:43903/ using document root <none> and password file <none>
I20260924 10:39:43.464223 3780 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:43.464293 3780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:43.464522 3780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:43.466679 3780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "d10887d398a64a2d82142a92ed3ca7f9"
format_stamp: "Formatted at 2026-09-24 10:39:43 on dist-test-slave-j38h"
server_key: "1ebde63bc85d598f5d6bda8a5910acde"
server_key_iv: "3ceee59a96bc59a70a0ac01b85734d7b"
server_key_version: "encryptionkey@0"
I20260924 10:39:43.467247 3780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "d10887d398a64a2d82142a92ed3ca7f9"
format_stamp: "Formatted at 2026-09-24 10:39:43 on dist-test-slave-j38h"
server_key: "1ebde63bc85d598f5d6bda8a5910acde"
server_key_iv: "3ceee59a96bc59a70a0ac01b85734d7b"
server_key_version: "encryptionkey@0"
I20260924 10:39:43.470956 3780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260924 10:39:43.473477 3795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:43.474768 3780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:39:43.474938 3780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "d10887d398a64a2d82142a92ed3ca7f9"
format_stamp: "Formatted at 2026-09-24 10:39:43 on dist-test-slave-j38h"
server_key: "1ebde63bc85d598f5d6bda8a5910acde"
server_key_iv: "3ceee59a96bc59a70a0ac01b85734d7b"
server_key_version: "encryptionkey@0"
I20260924 10:39:43.475061 3780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:43.485814 3780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:43.488711 3780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:43.488903 3780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:43.496637 3780 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:43213
I20260924 10:39:43.496650 3857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:43213 every 8 connection(s)
I20260924 10:39:43.497596 3780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:43.499887 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 3780
I20260924 10:39:43.500020 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:39:43.500301 2080 external_mini_cluster.cc:1428] Setting key 3497cc11e27773a57741f0a0733a86f4
I20260924 10:39:43.500571 3858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:43.508028 3858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9: Bootstrap starting.
Sep 24 10:39:43 dist-test-slave-j38h krb5kdc[3767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:43.510581 3858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:43.511854 3858 log.cc:824] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:43.513759 3858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9: No bootstrap required, opened a new log
I20260924 10:39:43.515108 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:43.501698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.2.8.62:43213)
0924 10:39:43.502096 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:43.502113 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:43.502154 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:39:43.502882 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:43.502904 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:43.503325 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:43.503682 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:43.504305 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:43.505811 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:43.506963 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:43.507700 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:43.507871 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:43.510610 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:43.510616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:43.510639 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:43.510665 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:43.512417 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:43.512837 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:43.512843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:43.512849 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:43.512965 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:43.513273 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:43.513276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:43.513278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:43.513657 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:43.513862 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:43.514142 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 10:39:43.514426 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":113,"threads_started":1}
I20260924 10:39:43.516798 3858 raft_consensus.cc:348] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10887d398a64a2d82142a92ed3ca7f9" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43213 } }
I20260924 10:39:43.517052 3858 raft_consensus.cc:374] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:43.517151 3858 raft_consensus.cc:729] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10887d398a64a2d82142a92ed3ca7f9, State: Initialized, Role: FOLLOWER
I20260924 10:39:43.517599 3858 consensus_queue.cc:260] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10887d398a64a2d82142a92ed3ca7f9" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43213 } }
I20260924 10:39:43.517728 3858 raft_consensus.cc:388] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:43.517767 3858 raft_consensus.cc:482] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:43.517831 3858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:43.518679 3858 raft_consensus.cc:504] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10887d398a64a2d82142a92ed3ca7f9" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43213 } }
I20260924 10:39:43.518956 3858 leader_election.cc:302] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d10887d398a64a2d82142a92ed3ca7f9; no voters:
I20260924 10:39:43.519191 3858 leader_election.cc:288] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:43.519327 3863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:43.519577 3863 raft_consensus.cc:686] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [term 1 LEADER]: Becoming Leader. State: Replica: d10887d398a64a2d82142a92ed3ca7f9, State: Running, Role: LEADER
I20260924 10:39:43.519927 3863 consensus_queue.cc:237] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10887d398a64a2d82142a92ed3ca7f9" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43213 } }
I20260924 10:39:43.520478 3858 sys_catalog.cc:564] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:43.521668 3864 sys_catalog.cc:455] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d10887d398a64a2d82142a92ed3ca7f9. Latest consensus state: current_term: 1 leader_uuid: "d10887d398a64a2d82142a92ed3ca7f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10887d398a64a2d82142a92ed3ca7f9" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43213 } } }
I20260924 10:39:43.521703 3865 sys_catalog.cc:455] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d10887d398a64a2d82142a92ed3ca7f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10887d398a64a2d82142a92ed3ca7f9" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43213 } } }
I20260924 10:39:43.521811 3864 sys_catalog.cc:458] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:43.521817 3865 sys_catalog.cc:458] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:43.522192 3872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:43.525059 3872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:43.530490 3872 catalog_manager.cc:1347] Generated new cluster ID: 445b0076a6f44913aa069462fd194397
I20260924 10:39:43.530572 3872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:43.549959 3872 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:43.551225 3872 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:43.565887 3872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9: Generated new TSK 0
I20260924 10:39:43.566757 3872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:39:43.606468 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43213
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:40333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:43.720161 3884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:43.720472 3884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:43.720552 3884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:43.724857 3884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:43.724915 3884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:43.724959 3884 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 10:39:43.724993 3884 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:39:43.725075 3884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:39:43.730120 3884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:40333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.8.62:43213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:43.731069 3884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:43.732023 3884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:43.741274 3890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:43.741283 3892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:43.741573 3889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:43.741742 3884 server_base.cc:1034] running on GCE node
I20260924 10:39:43.742112 3884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:43.742733 3884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:43.743973 3884 hybrid_clock.cc:648] HybridClock initialized: now 1790246383743943 us; error 44 us; skew 500 ppm
Sep 24 10:39:43 dist-test-slave-j38h krb5kdc[3767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:43.746891 3884 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:39:43.747745 3884 webserver.cc:460] Webserver started at http://127.2.8.1:38677/ using document root <none> and password file <none>
I20260924 10:39:43.748289 3884 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:43.748422 3884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:43.748647 3884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:43.750669 3884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "8a574b056885428d87c58bc279a93d46"
format_stamp: "Formatted at 2026-09-24 10:39:43 on dist-test-slave-j38h"
server_key: "75eb1d6c8c534a585fbce2a5f3fce2d9"
server_key_iv: "8b556187fd31a0a8e005c44a542d1d74"
server_key_version: "encryptionkey@0"
I20260924 10:39:43.751209 3884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "8a574b056885428d87c58bc279a93d46"
format_stamp: "Formatted at 2026-09-24 10:39:43 on dist-test-slave-j38h"
server_key: "75eb1d6c8c534a585fbce2a5f3fce2d9"
server_key_iv: "8b556187fd31a0a8e005c44a542d1d74"
server_key_version: "encryptionkey@0"
I20260924 10:39:43.754853 3884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:39:43.757301 3899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:43.758493 3884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:39:43.758631 3884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "8a574b056885428d87c58bc279a93d46"
format_stamp: "Formatted at 2026-09-24 10:39:43 on dist-test-slave-j38h"
server_key: "75eb1d6c8c534a585fbce2a5f3fce2d9"
server_key_iv: "8b556187fd31a0a8e005c44a542d1d74"
server_key_version: "encryptionkey@0"
I20260924 10:39:43.758744 3884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:43.769848 3884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:43.773165 3884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:43.773387 3884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:43.774858 3884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:43.774936 3884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:43.775010 3884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:43.775064 3884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:39:43 dist-test-slave-j38h krb5kdc[3767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:43.786469 3884 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:45995
I20260924 10:39:43.786513 4014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:45995 every 8 connection(s)
I20260924 10:39:43.787784 3884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:39:43.789649 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:43.775551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:40219 (local address 127.2.8.62:43213)
0924 10:39:43.775732 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:43.775738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:43.776905 (+ 1167us) server_negotiation.cc:407] Connection header received
0924 10:39:43.778085 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:43.778089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:43.778142 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:43.778226 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:43.779698 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:43.780412 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:43.781736 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:43.781983 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:43.782100 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:43.785507 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:43.785510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:43.785512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:43.785537 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:43.787537 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:43.788019 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:43.788023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:43.788024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:43.788121 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:43.788569 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:43.788572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:43.788574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:43.788802 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:43.788882 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:43.789401 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 10:39:43.789528 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 10:39:43.790433 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:43.776336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43213 (local address 127.2.8.1:40219)
0924 10:39:43.776757 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:43.776804 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:43.777874 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:43.778426 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:43.778433 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:43.778907 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:43.779512 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:43.779531 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:43.780541 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:43.780544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:43.781611 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:43.781620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:43.782102 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:43.782154 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:43.782326 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:43.783002 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:43.783021 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:43.785336 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:43.787655 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:43.787660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:43.787671 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:43.787922 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:43.788243 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:43.788250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:43.788253 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:43.788415 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:43.788908 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:43.788913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:43.789187 (+ 274us) client_negotiation.cc:769] Sending connection context
0924 10:39:43.789333 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 10:39:43.789582 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":128,"threads_started":1}
I20260924 10:39:43.791610 4015 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43213
I20260924 10:39:43.791736 4015 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:43.792274 4015 heartbeater.cc:511] Master 127.2.8.62:43213 requested a full tablet report, sending...
I20260924 10:39:43.792809 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 3884
I20260924 10:39:43.793004 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:39:43.793382 2080 external_mini_cluster.cc:1428] Setting key 5fc13746a67960727596c88fd9d6c8f3
I20260924 10:39:43.794076 3811 ts_manager.cc:194] Registered new tserver with Master: 8a574b056885428d87c58bc279a93d46 (127.2.8.1:45995)
I20260924 10:39:43.795683 3811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:40219
I20260924 10:39:43.801779 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:43.794101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:60509 (local address 127.2.8.62:43213)
0924 10:39:43.794222 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:43.794226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:43.794456 (+ 230us) server_negotiation.cc:407] Connection header received
0924 10:39:43.794543 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:43.794546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:43.794589 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:43.794680 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:43.795162 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:43.795669 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:43.796518 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:43.796825 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:43.796935 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:43.798391 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:43.798394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:43.798396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:43.798418 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:43.800022 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:43.800536 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:43.800539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:43.800540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:43.800614 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:43.800931 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:43.800934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:43.800935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:43.801153 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:43.801234 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:43.801561 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 10:39:43.801659 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:39:43.801889 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:43.794019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43213 (local address 127.2.8.1:60509)
0924 10:39:43.794356 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:43.794371 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:43.794465 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:43.794699 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:43.794702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:43.794911 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:43.795040 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:43.795047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:43.795828 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:43.795831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:43.796401 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:43.796408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:43.797056 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:43.797075 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:43.797163 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:43.797771 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:43.797785 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:43.798203 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:43.800135 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:43.800140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:43.800142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:43.800432 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:43.800716 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:43.800723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:43.800728 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:43.800834 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:43.801252 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:43.801258 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:43.801382 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 10:39:43.801504 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 10:39:43.801653 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:39:43.822774 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43213
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:40333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:43.932080 4024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:43.932343 4024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:43.932390 4024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:43.936383 4024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:43.936443 4024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:43.936483 4024 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 10:39:43.936497 4024 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:39:43.936574 4024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:39:43.941859 4024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:40333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.8.62:43213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:43.942926 4024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:43.943933 4024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:43.953624 4032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:43.953577 4030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:43.953718 4024 server_base.cc:1034] running on GCE node
W20260924 10:39:43.953577 4029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:43.954300 4024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:43.954984 4024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:43.956152 4024 hybrid_clock.cc:648] HybridClock initialized: now 1790246383956130 us; error 33 us; skew 500 ppm
Sep 24 10:39:43 dist-test-slave-j38h krb5kdc[3767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:43.958896 4024 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:39:43.959810 4024 webserver.cc:460] Webserver started at http://127.2.8.2:45597/ using document root <none> and password file <none>
I20260924 10:39:43.960333 4024 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:43.960391 4024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:43.960551 4024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:43.962647 4024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "49e56cd06f5a464cb2b2a1f3332ced97"
format_stamp: "Formatted at 2026-09-24 10:39:43 on dist-test-slave-j38h"
server_key: "15835223a8b78c6cbbb607f20873ab42"
server_key_iv: "c0f0e3dec64b4a8ade2b5365f446d02a"
server_key_version: "encryptionkey@0"
I20260924 10:39:43.963183 4024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "49e56cd06f5a464cb2b2a1f3332ced97"
format_stamp: "Formatted at 2026-09-24 10:39:43 on dist-test-slave-j38h"
server_key: "15835223a8b78c6cbbb607f20873ab42"
server_key_iv: "c0f0e3dec64b4a8ade2b5365f446d02a"
server_key_version: "encryptionkey@0"
I20260924 10:39:43.967034 4024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:39:43.969424 4039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:43.970620 4024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:39:43.970759 4024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "49e56cd06f5a464cb2b2a1f3332ced97"
format_stamp: "Formatted at 2026-09-24 10:39:43 on dist-test-slave-j38h"
server_key: "15835223a8b78c6cbbb607f20873ab42"
server_key_iv: "c0f0e3dec64b4a8ade2b5365f446d02a"
server_key_version: "encryptionkey@0"
I20260924 10:39:43.970882 4024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:43.985172 4024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:43.987969 4024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:43.988169 4024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:43.989634 4024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:43.989703 4024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:43.989771 4024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:43.989823 4024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:39:43 dist-test-slave-j38h krb5kdc[3767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:44.000339 4024 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:34367
I20260924 10:39:44.000427 4154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:34367 every 8 connection(s)
I20260924 10:39:44.001329 4024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:39:44.004091 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:43.990302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:44237 (local address 127.2.8.62:43213)
0924 10:39:43.990455 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:43.990460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:43.991839 (+ 1379us) server_negotiation.cc:407] Connection header received
0924 10:39:43.993360 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:43.993366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:43.993428 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:43.993538 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:43.994741 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:43.995635 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:43.997122 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:43.997395 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:43.997480 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.000550 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:44.000555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:44.000557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:44.000582 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:44.002219 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:44.002716 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:44.002719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:44.002721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:44.002796 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:44.003143 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:44.003145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:44.003146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:44.003311 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:44.003397 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:44.003878 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 10:39:44.003975 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:39:44.005055 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:43.991178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43213 (local address 127.2.8.2:44237)
0924 10:39:43.991684 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:43.991739 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:43.993116 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:43.993822 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:43.993836 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:43.994135 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:43.994595 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:43.994608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:43.995773 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:43.995776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:43.996984 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:43.997017 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:43.997513 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:43.997564 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:43.997804 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:43.998447 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:43.998474 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:44.000399 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:44.002332 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:44.002338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:44.002351 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:44.002612 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:44.002899 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:44.002902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:44.002903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:44.003050 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:44.003424 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:44.003429 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:44.003660 (+ 231us) client_negotiation.cc:769] Sending connection context
0924 10:39:44.003827 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 10:39:44.004071 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":142,"threads_started":1}
I20260924 10:39:44.006268 4155 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43213
I20260924 10:39:44.006434 4155 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:44.006923 4155 heartbeater.cc:511] Master 127.2.8.62:43213 requested a full tablet report, sending...
I20260924 10:39:44.007907 3812 ts_manager.cc:194] Registered new tserver with Master: 49e56cd06f5a464cb2b2a1f3332ced97 (127.2.8.2:34367)
I20260924 10:39:44.008596 3812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:44237
I20260924 10:39:44.008894 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 4024
I20260924 10:39:44.009052 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:39:44.009407 2080 external_mini_cluster.cc:1428] Setting key 3fa97809829da646919c2dd822598168
I20260924 10:39:44.016503 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.008525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:44155 (local address 127.2.8.62:43213)
0924 10:39:44.008649 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:44.008656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:44.008822 (+ 166us) server_negotiation.cc:407] Connection header received
0924 10:39:44.009019 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:44.009022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:44.009072 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:44.009128 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:44.009754 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.010490 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.011367 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.011759 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.011881 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.013175 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:44.013179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:44.013182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:44.013205 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:44.014864 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:44.015283 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:44.015288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:44.015289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:44.015372 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:44.015654 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:44.015657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:44.015659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:44.015872 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:44.015981 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:44.016231 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 10:39:44.016336 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:39:44.016516 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.008369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43213 (local address 127.2.8.2:44155)
0924 10:39:44.008735 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:44.008752 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:44.008888 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:44.009272 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:44.009275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:44.009497 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:44.009631 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.009639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.010639 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.010648 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.011242 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.011251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.011886 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.011902 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.011982 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.012551 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:44.012563 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:44.012968 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:44.014983 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:44.014986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:44.014988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:44.015187 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:44.015475 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:44.015478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:44.015480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:44.015562 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:44.015971 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:44.015974 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:44.016073 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 10:39:44.016178 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 10:39:44.016324 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:39:44.034241 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43213
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:40333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:44.143138 4164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:44.143404 4164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:44.143450 4164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:44.148085 4164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:44.148149 4164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:44.148237 4164 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 10:39:44.148260 4164 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:39:44.148308 4164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:39:44.153661 4164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:40333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.8.62:43213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:44.154688 4164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:44.155652 4164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:44.165372 4170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:44.165334 4172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:44.165536 4164 server_base.cc:1034] running on GCE node
W20260924 10:39:44.165344 4169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:44.166020 4164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:44.166651 4164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:44.167841 4164 hybrid_clock.cc:648] HybridClock initialized: now 1790246384167800 us; error 49 us; skew 500 ppm
Sep 24 10:39:44 dist-test-slave-j38h krb5kdc[3767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246384, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:44.170814 4164 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:39:44.171685 4164 webserver.cc:460] Webserver started at http://127.2.8.3:44387/ using document root <none> and password file <none>
I20260924 10:39:44.172240 4164 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:44.172328 4164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:44.172533 4164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:44.174690 4164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "1b0aadae619d4689a5251f9ab93dce21"
format_stamp: "Formatted at 2026-09-24 10:39:44 on dist-test-slave-j38h"
server_key: "dc405295710676b1ab0b83fb1dfc67fe"
server_key_iv: "849c3f20cec010915eab864aa3591066"
server_key_version: "encryptionkey@0"
I20260924 10:39:44.175246 4164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "1b0aadae619d4689a5251f9ab93dce21"
format_stamp: "Formatted at 2026-09-24 10:39:44 on dist-test-slave-j38h"
server_key: "dc405295710676b1ab0b83fb1dfc67fe"
server_key_iv: "849c3f20cec010915eab864aa3591066"
server_key_version: "encryptionkey@0"
I20260924 10:39:44.178915 4164 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 10:39:44.181465 4179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:44.182652 4164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:39:44.182792 4164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "1b0aadae619d4689a5251f9ab93dce21"
format_stamp: "Formatted at 2026-09-24 10:39:44 on dist-test-slave-j38h"
server_key: "dc405295710676b1ab0b83fb1dfc67fe"
server_key_iv: "849c3f20cec010915eab864aa3591066"
server_key_version: "encryptionkey@0"
I20260924 10:39:44.182906 4164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:44.200161 4164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:44.203495 4164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:44.203696 4164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:44.205109 4164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:44.205175 4164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:44.205240 4164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:44.205289 4164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:39:44 dist-test-slave-j38h krb5kdc[3767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246384, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:44.216260 4164 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:40493
I20260924 10:39:44.216324 4294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:40493 every 8 connection(s)
I20260924 10:39:44.217208 4164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:39:44.220127 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.206099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:58673 (local address 127.2.8.62:43213)
0924 10:39:44.206230 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:44.206234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:44.207375 (+ 1141us) server_negotiation.cc:407] Connection header received
0924 10:39:44.208518 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:44.208521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:44.208580 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:44.208687 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:44.210116 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.210889 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.212124 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.212441 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.212548 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.216480 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:44.216484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:44.216487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:44.216523 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:44.218163 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:44.218688 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:44.218692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:44.218693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:44.218797 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:44.219094 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:44.219096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:44.219097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:44.219277 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:44.219356 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:44.219818 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 10:39:44.219937 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:39:44.220170 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 4164
I20260924 10:39:44.220275 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:39:44.220561 2080 external_mini_cluster.cc:1428] Setting key f66a78bf5b2c5c9b8121a9d137d64dd4
I20260924 10:39:44.221153 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.206644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43213 (local address 127.2.8.3:58673)
0924 10:39:44.207181 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:44.207232 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:44.208304 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:44.208863 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:44.208870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:44.209328 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:44.209977 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.209992 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.211034 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.211038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.211999 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.212013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.212603 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.212667 (+ 64us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.212915 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.214305 (+ 1390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:44.214327 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:44.216258 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:44.218270 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:44.218274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:44.218284 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:44.218572 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:44.218888 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:44.218890 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:44.218892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:44.219013 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:44.219390 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:44.219395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:44.219610 (+ 215us) client_negotiation.cc:769] Sending connection context
0924 10:39:44.219759 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 10:39:44.220101 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":154,"threads_started":1}
I20260924 10:39:44.222503 4295 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43213
I20260924 10:39:44.222643 4295 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:44.223114 4295 heartbeater.cc:511] Master 127.2.8.62:43213 requested a full tablet report, sending...
I20260924 10:39:44.224138 3812 ts_manager.cc:194] Registered new tserver with Master: 1b0aadae619d4689a5251f9ab93dce21 (127.2.8.3:40493)
I20260924 10:39:44.224881 3812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:58673
I20260924 10:39:44.231703 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.224599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:53549 (local address 127.2.8.62:43213)
0924 10:39:44.224725 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:44.224729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:44.224862 (+ 133us) server_negotiation.cc:407] Connection header received
0924 10:39:44.225031 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:44.225035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:44.225083 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:44.225187 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:44.225638 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.226188 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.227068 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.227325 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.227450 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.228577 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:44.228581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:44.228584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:44.228607 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:44.230028 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:44.230490 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:44.230495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:44.230497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:44.230582 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:44.230878 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:44.230884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:44.230886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:44.231060 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:44.231173 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:44.231441 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 10:39:44.231554 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:39:44.231715 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.224482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43213 (local address 127.2.8.3:53549)
0924 10:39:44.224755 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:44.224773 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:44.224906 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:44.225199 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:44.225202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:44.225423 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:44.225520 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.225526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.226315 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.226319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.226935 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.226943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.227436 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.227453 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.227557 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.228104 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:44.228117 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:44.228429 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:44.230144 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:44.230150 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:44.230153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:44.230384 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:44.230682 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:44.230687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:44.230690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:44.230776 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:44.231163 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:44.231167 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:44.231269 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 10:39:44.231384 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 10:39:44.231527 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":105,"threads_started":1}
I20260924 10:39:44.234484 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:39:44 dist-test-slave-j38h krb5kdc[3767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:39:44 dist-test-slave-j38h krb5kdc[3767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:44.259788 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.251940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.2.8.62:43213)
0924 10:39:44.252077 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:44.252080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:44.252094 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:39:44.252129 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:44.252131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:44.252167 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:44.252241 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:44.253078 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.253631 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.254447 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.254713 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.254803 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.257102 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:44.257105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:44.257107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:44.257128 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:44.258450 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:44.258815 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:44.258817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:44.258818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:44.258887 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:44.259103 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:44.259105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:44.259106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:44.259262 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:44.259401 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:44.259594 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 10:39:44.259678 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:39:44.262575 3812 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:39:44.265688 3812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:39:44.280158 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.273725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:34367 (local address 127.0.0.1:51004)
0924 10:39:44.274673 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:44.274723 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:44.274861 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:44.275638 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:44.275641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:44.275658 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:44.275785 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.275790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.277253 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.277256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.278343 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.278351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.279574 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.279577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.279680 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.279728 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 10:39:44.279839 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 10:39:44.279953 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":45,"thread_start_us":80,"threads_started":1}
I20260924 10:39:44.280557 4313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.274479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:40493 (local address 127.0.0.1:48772)
0924 10:39:44.275866 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:44.275880 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:44.275964 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:44.276373 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:44.276376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:44.276393 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:44.276474 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.276479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.277796 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.277800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.278994 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.279001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.280156 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.280161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.280260 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.280307 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 10:39:44.280368 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 10:39:44.280428 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1317,"mutex_wait_us":64,"thread_start_us":73,"threads_started":1}
I20260924 10:39:44.280678 4317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.274760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51004 (local address 127.2.8.2:34367)
0924 10:39:44.275223 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:44.275235 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:44.275262 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:39:44.275331 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:44.275334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:44.275478 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:44.275642 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:44.275905 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.277124 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.278467 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.279226 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.279354 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.279512 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:44.280459 (+ 947us) server_negotiation.cc:299] Negotiation successful
0924 10:39:44.280537 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":109,"threads_started":1}
I20260924 10:39:44.281029 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.274622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48772 (local address 127.2.8.3:40493)
0924 10:39:44.275078 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:44.275088 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:44.275969 (+ 881us) server_negotiation.cc:407] Connection header received
0924 10:39:44.276069 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:44.276073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:44.276237 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:44.276373 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:44.276588 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.277673 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.279099 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.279822 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.279916 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.280089 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:44.280764 (+ 675us) server_negotiation.cc:299] Negotiation successful
0924 10:39:44.280872 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":114,"threads_started":1}
I20260924 10:39:44.281301 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.273944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.2.8.1:45995)
0924 10:39:44.274440 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:44.274458 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:44.275019 (+ 561us) server_negotiation.cc:407] Connection header received
0924 10:39:44.275229 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:44.275234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:44.275434 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:44.275554 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:44.275929 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.277185 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.278890 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.279676 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.279783 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.279999 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:44.280201 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 10:39:44.281011 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":90,"threads_started":1}
I20260924 10:39:44.281689 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.273762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:45995 (local address 127.0.0.1:39914)
0924 10:39:44.274924 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:44.274937 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:44.275092 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:44.275585 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:44.275594 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:44.275619 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:44.275784 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.275793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.277332 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.277335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.278770 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.278780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.279806 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.279810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.279909 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.279933 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:39:44.281500 (+ 1567us) client_negotiation.cc:240] Negotiation successful
0924 10:39:44.281572 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"mutex_wait_us":115,"thread_start_us":80,"threads_started":1}
I20260924 10:39:44.283238 4089 tablet_service.cc:1467] Processing CreateTablet for tablet e3735b746dd6443a96d229bcff064ef7 (DEFAULT_TABLE table=test-table [id=b804c8f2cf7047c4a8675ab8cb78312b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:44.283562 4229 tablet_service.cc:1467] Processing CreateTablet for tablet e3735b746dd6443a96d229bcff064ef7 (DEFAULT_TABLE table=test-table [id=b804c8f2cf7047c4a8675ab8cb78312b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:44.284212 4089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3735b746dd6443a96d229bcff064ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:44.283941 3949 tablet_service.cc:1467] Processing CreateTablet for tablet e3735b746dd6443a96d229bcff064ef7 (DEFAULT_TABLE table=test-table [id=b804c8f2cf7047c4a8675ab8cb78312b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:44.284565 4229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3735b746dd6443a96d229bcff064ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:44.284821 3949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3735b746dd6443a96d229bcff064ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:44.290802 4318 tablet_bootstrap.cc:492] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97: Bootstrap starting.
I20260924 10:39:44.292459 4319 tablet_bootstrap.cc:492] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46: Bootstrap starting.
I20260924 10:39:44.293184 4318 tablet_bootstrap.cc:654] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:44.293306 4320 tablet_bootstrap.cc:492] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21: Bootstrap starting.
I20260924 10:39:44.293936 4318 log.cc:824] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:44.294477 4319 tablet_bootstrap.cc:654] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:44.295169 4320 tablet_bootstrap.cc:654] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:44.295248 4319 log.cc:824] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:44.295862 4320 log.cc:824] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:44.296119 4318 tablet_bootstrap.cc:492] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97: No bootstrap required, opened a new log
I20260924 10:39:44.296310 4318 ts_tablet_manager.cc:1397] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:39:44.297219 4319 tablet_bootstrap.cc:492] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46: No bootstrap required, opened a new log
I20260924 10:39:44.297399 4319 ts_tablet_manager.cc:1397] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:39:44.297560 4320 tablet_bootstrap.cc:492] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21: No bootstrap required, opened a new log
I20260924 10:39:44.297736 4320 ts_tablet_manager.cc:1397] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:39:44.300216 4318 raft_consensus.cc:348] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.300485 4318 raft_consensus.cc:374] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:44.300334 4320 raft_consensus.cc:348] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.300541 4318 raft_consensus.cc:729] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e56cd06f5a464cb2b2a1f3332ced97, State: Initialized, Role: FOLLOWER
I20260924 10:39:44.300551 4320 raft_consensus.cc:374] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:44.300611 4320 raft_consensus.cc:729] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0aadae619d4689a5251f9ab93dce21, State: Initialized, Role: FOLLOWER
I20260924 10:39:44.300964 4318 consensus_queue.cc:260] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.301012 4320 consensus_queue.cc:260] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.301010 4319 raft_consensus.cc:348] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.301231 4319 raft_consensus.cc:374] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:44.301282 4319 raft_consensus.cc:729] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a574b056885428d87c58bc279a93d46, State: Initialized, Role: FOLLOWER
I20260924 10:39:44.301621 4319 consensus_queue.cc:260] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.301937 4318 ts_tablet_manager.cc:1428] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260924 10:39:44.301937 4320 ts_tablet_manager.cc:1428] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 10:39:44.302263 4155 heartbeater.cc:503] Master 127.2.8.62:43213 was elected leader, sending a full tablet report...
I20260924 10:39:44.302452 4319 ts_tablet_manager.cc:1428] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:39:44.302654 4015 heartbeater.cc:503] Master 127.2.8.62:43213 was elected leader, sending a full tablet report...
I20260924 10:39:44.303267 4295 heartbeater.cc:503] Master 127.2.8.62:43213 was elected leader, sending a full tablet report...
W20260924 10:39:44.468204 4296 tablet.cc:2369] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:39:44.502700 4156 tablet.cc:2369] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:39:44.539234 4016 tablet.cc:2369] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:39:44.555660 4325 raft_consensus.cc:482] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:39:44.555881 4325 raft_consensus.cc:504] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.557088 4325 leader_election.cc:288] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a574b056885428d87c58bc279a93d46 (127.2.8.1:45995), 1b0aadae619d4689a5251f9ab93dce21 (127.2.8.3:40493)
I20260924 10:39:44.561267 4327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.557379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:45995 (local address 127.2.8.2:52251)
0924 10:39:44.557688 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:44.557705 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:44.557865 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:44.558180 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:44.558183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:44.558201 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:44.558330 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.558336 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.559063 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.559069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.559935 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.559944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.560805 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.560809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.560908 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.560940 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:39:44.561038 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 10:39:44.561118 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":49,"thread_start_us":94,"threads_started":1}
I20260924 10:39:44.561281 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.557485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:52251 (local address 127.2.8.1:45995)
0924 10:39:44.557665 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:44.557670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:44.557806 (+ 136us) server_negotiation.cc:407] Connection header received
0924 10:39:44.558007 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:44.558011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:44.558072 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:44.558168 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:44.558445 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.558955 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.560084 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.560680 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.560786 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.560863 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:44.561096 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 10:39:44.561167 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 10:39:44.561470 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.557972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:43375 (local address 127.2.8.3:40493)
0924 10:39:44.558096 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:44.558101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:44.558156 (+ 55us) server_negotiation.cc:407] Connection header received
0924 10:39:44.558276 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:44.558279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:44.558326 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:44.558408 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:44.558663 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.559148 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.560211 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.560798 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.560906 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.560958 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:44.561212 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 10:39:44.561291 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:39:44.561829 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.557825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:40493 (local address 127.2.8.2:43375)
0924 10:39:44.558094 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:44.558108 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:44.558192 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:44.558418 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:44.558421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:44.558435 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:44.558549 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.558560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.559312 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.559315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.560093 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.560103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.560934 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.560938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.561071 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.561094 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:39:44.561610 (+ 516us) client_negotiation.cc:240] Negotiation successful
0924 10:39:44.561690 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
I20260924 10:39:44.562184 3969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3735b746dd6443a96d229bcff064ef7" candidate_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a574b056885428d87c58bc279a93d46" is_pre_election: true
I20260924 10:39:44.562582 3969 raft_consensus.cc:2454] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e56cd06f5a464cb2b2a1f3332ced97 in term 0.
I20260924 10:39:44.562573 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3735b746dd6443a96d229bcff064ef7" candidate_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0aadae619d4689a5251f9ab93dce21" is_pre_election: true
I20260924 10:39:44.562830 4249 raft_consensus.cc:2454] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e56cd06f5a464cb2b2a1f3332ced97 in term 0.
I20260924 10:39:44.563076 4042 leader_election.cc:302] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e56cd06f5a464cb2b2a1f3332ced97, 8a574b056885428d87c58bc279a93d46; no voters:
I20260924 10:39:44.563475 4325 raft_consensus.cc:2781] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:39:44.563563 4325 raft_consensus.cc:482] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:39:44.563616 4325 raft_consensus.cc:3037] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:44.564652 4325 raft_consensus.cc:504] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.565026 4325 leader_election.cc:288] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [CANDIDATE]: Term 1 election: Requested vote from peers 8a574b056885428d87c58bc279a93d46 (127.2.8.1:45995), 1b0aadae619d4689a5251f9ab93dce21 (127.2.8.3:40493)
I20260924 10:39:44.565423 3969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3735b746dd6443a96d229bcff064ef7" candidate_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a574b056885428d87c58bc279a93d46"
I20260924 10:39:44.565433 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3735b746dd6443a96d229bcff064ef7" candidate_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0aadae619d4689a5251f9ab93dce21"
I20260924 10:39:44.565528 3969 raft_consensus.cc:3037] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:44.565563 4249 raft_consensus.cc:3037] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:44.566664 3969 raft_consensus.cc:2454] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e56cd06f5a464cb2b2a1f3332ced97 in term 1.
I20260924 10:39:44.566951 4042 leader_election.cc:302] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e56cd06f5a464cb2b2a1f3332ced97, 8a574b056885428d87c58bc279a93d46; no voters:
I20260924 10:39:44.567044 4249 raft_consensus.cc:2454] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e56cd06f5a464cb2b2a1f3332ced97 in term 1.
I20260924 10:39:44.567098 4325 raft_consensus.cc:2781] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:44.567272 4325 raft_consensus.cc:686] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 LEADER]: Becoming Leader. State: Replica: 49e56cd06f5a464cb2b2a1f3332ced97, State: Running, Role: LEADER
I20260924 10:39:44.567641 4325 consensus_queue.cc:237] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.570607 3812 catalog_manager.cc:5495] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e56cd06f5a464cb2b2a1f3332ced97 (127.2.8.2). New cstate: current_term: 1 leader_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } health_report { overall_health: UNKNOWN } } }
Sep 24 10:39:44 dist-test-slave-j38h krb5kdc[3767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246383, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:44.658528 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.650814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43213 (local address 127.0.0.1:41974)
0924 10:39:44.651113 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:44.651125 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:44.651224 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:44.651454 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:44.651456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:44.651868 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:44.651962 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.651968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.652683 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.652686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.653315 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.653325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.653709 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.653722 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.653795 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.654374 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:44.654386 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:44.655749 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:44.657246 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:44.657253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:44.657258 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:44.657470 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:44.657689 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:44.657691 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:44.657692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:44.657763 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:44.658069 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:44.658074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:44.658164 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 10:39:44.658250 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 10:39:44.658363 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":127,"threads_started":1}
I20260924 10:39:44.658543 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.650880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.2.8.62:43213)
0924 10:39:44.651013 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:44.651019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:44.651197 (+ 178us) server_negotiation.cc:407] Connection header received
0924 10:39:44.651301 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:44.651304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:44.651348 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:44.651417 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:44.652068 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.652582 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.653423 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.653627 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.653708 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.655873 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:44.655876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:44.655878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:44.655897 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:44.657150 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:44.657548 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:44.657550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:44.657552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:44.657617 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:44.657840 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:44.657842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:44.657843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:44.657988 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:44.658059 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:44.658301 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 10:39:44.658426 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":52}
I20260924 10:39:44.662508 3812 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:41974:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 10:39:44.663270 3812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:39:44.668324 4229 tablet_service.cc:1467] Processing CreateTablet for tablet cbcaf14491694c21a40d7d745fc2f5db (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c8e51d1109544476ac27ff7d968f8e38]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:39:44.668144 3949 tablet_service.cc:1467] Processing CreateTablet for tablet cbcaf14491694c21a40d7d745fc2f5db (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c8e51d1109544476ac27ff7d968f8e38]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:39:44.668202 4089 tablet_service.cc:1467] Processing CreateTablet for tablet cbcaf14491694c21a40d7d745fc2f5db (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c8e51d1109544476ac27ff7d968f8e38]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:39:44.668913 4229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbcaf14491694c21a40d7d745fc2f5db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:44.668913 3949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbcaf14491694c21a40d7d745fc2f5db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:44.668915 4089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbcaf14491694c21a40d7d745fc2f5db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:44.671589 4318 tablet_bootstrap.cc:492] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97: Bootstrap starting.
I20260924 10:39:44.671682 4320 tablet_bootstrap.cc:492] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21: Bootstrap starting.
I20260924 10:39:44.672530 4320 tablet_bootstrap.cc:654] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:44.672696 4318 tablet_bootstrap.cc:654] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:44.673393 4319 tablet_bootstrap.cc:492] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46: Bootstrap starting.
I20260924 10:39:44.674098 4318 tablet_bootstrap.cc:492] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97: No bootstrap required, opened a new log
I20260924 10:39:44.674165 4318 ts_tablet_manager.cc:1397] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:39:44.674235 4319 tablet_bootstrap.cc:654] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:44.674679 4318 raft_consensus.cc:348] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.674809 4318 raft_consensus.cc:374] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:44.674849 4318 raft_consensus.cc:729] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e56cd06f5a464cb2b2a1f3332ced97, State: Initialized, Role: FOLLOWER
I20260924 10:39:44.675006 4318 consensus_queue.cc:260] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.675199 4320 tablet_bootstrap.cc:492] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21: No bootstrap required, opened a new log
I20260924 10:39:44.675294 4320 ts_tablet_manager.cc:1397] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260924 10:39:44.675310 4325 tablet_replica.cc:439] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } } }
I20260924 10:39:44.675455 4325 tablet_replica.cc:442] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:39:44.675781 4320 raft_consensus.cc:348] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.675894 4320 raft_consensus.cc:374] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:44.675935 4320 raft_consensus.cc:729] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0aadae619d4689a5251f9ab93dce21, State: Initialized, Role: FOLLOWER
I20260924 10:39:44.676056 4320 consensus_queue.cc:260] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.676095 4318 ts_tablet_manager.cc:1428] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:39:44.676306 4320 ts_tablet_manager.cc:1428] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:39:44.676522 4324 tablet_replica.cc:439] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } } }
I20260924 10:39:44.676643 4324 tablet_replica.cc:442] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:39:44.677219 4319 tablet_bootstrap.cc:492] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46: No bootstrap required, opened a new log
I20260924 10:39:44.677305 4319 ts_tablet_manager.cc:1397] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 10:39:44.677769 4319 raft_consensus.cc:348] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.677877 4319 raft_consensus.cc:374] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:44.677910 4319 raft_consensus.cc:729] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a574b056885428d87c58bc279a93d46, State: Initialized, Role: FOLLOWER
I20260924 10:39:44.678035 4319 consensus_queue.cc:260] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.678329 4319 ts_tablet_manager.cc:1428] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:39:44.678576 4326 tablet_replica.cc:439] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } } }
I20260924 10:39:44.678707 4326 tablet_replica.cc:442] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:39:44.955040 4325 consensus_queue.cc:1035] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:44.960813 4325 consensus_queue.cc:1035] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:44.973451 4324 raft_consensus.cc:482] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:39:44.973562 4324 raft_consensus.cc:504] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.974650 4324 leader_election.cc:288] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a574b056885428d87c58bc279a93d46 (127.2.8.1:45995), 49e56cd06f5a464cb2b2a1f3332ced97 (127.2.8.2:34367)
I20260924 10:39:44.978571 4346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.974906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:34367 (local address 127.2.8.3:55425)
0924 10:39:44.975226 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:44.975240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:44.975369 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:44.975663 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:44.975665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:44.975679 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:44.975798 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.975803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.976542 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.976545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.977498 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.977507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.978135 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.978139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.978242 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.978270 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:39:44.978364 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 10:39:44.978431 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":45,"thread_start_us":85,"threads_started":1}
I20260924 10:39:44.978570 4348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.975070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:55425 (local address 127.2.8.2:34367)
0924 10:39:44.975341 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:44.975351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:44.975371 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:39:44.975490 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:44.975496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:44.975552 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:44.975655 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:44.975904 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.976428 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.977619 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.978020 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.978133 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.978191 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:44.978371 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 10:39:44.978439 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
I20260924 10:39:44.978595 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.974943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:48969 (local address 127.2.8.1:45995)
0924 10:39:44.975084 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:44.975088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:44.975428 (+ 340us) server_negotiation.cc:407] Connection header received
0924 10:39:44.975571 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:44.975573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:44.975607 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:44.975691 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:44.975935 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.976455 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.977613 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.978008 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.978124 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.978190 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:44.978403 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 10:39:44.978475 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:39:44.978616 4347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:44.974911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:45995 (local address 127.2.8.3:48969)
0924 10:39:44.975346 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:44.975371 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:44.975470 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:44.975700 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:44.975704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:44.975719 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:44.975830 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.975835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.976562 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.976568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.977494 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:44.977503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:44.978168 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:44.978172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:44.978270 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:44.978287 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:39:44.978416 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 10:39:44.978479 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":35,"thread_start_us":67,"threads_started":1}
I20260924 10:39:44.979097 3969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbcaf14491694c21a40d7d745fc2f5db" candidate_uuid: "1b0aadae619d4689a5251f9ab93dce21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a574b056885428d87c58bc279a93d46" is_pre_election: true
I20260924 10:39:44.979282 3969 raft_consensus.cc:2454] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b0aadae619d4689a5251f9ab93dce21 in term 0.
I20260924 10:39:44.979533 4091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbcaf14491694c21a40d7d745fc2f5db" candidate_uuid: "1b0aadae619d4689a5251f9ab93dce21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" is_pre_election: true
I20260924 10:39:44.979607 4182 leader_election.cc:302] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0aadae619d4689a5251f9ab93dce21, 8a574b056885428d87c58bc279a93d46; no voters:
I20260924 10:39:44.979748 4091 raft_consensus.cc:2454] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b0aadae619d4689a5251f9ab93dce21 in term 0.
I20260924 10:39:44.979918 4324 raft_consensus.cc:2781] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:39:44.979987 4324 raft_consensus.cc:482] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:39:44.980042 4324 raft_consensus.cc:3037] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:44.981091 4324 raft_consensus.cc:504] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.981418 4324 leader_election.cc:288] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [CANDIDATE]: Term 1 election: Requested vote from peers 8a574b056885428d87c58bc279a93d46 (127.2.8.1:45995), 49e56cd06f5a464cb2b2a1f3332ced97 (127.2.8.2:34367)
I20260924 10:39:44.981735 4091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbcaf14491694c21a40d7d745fc2f5db" candidate_uuid: "1b0aadae619d4689a5251f9ab93dce21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e56cd06f5a464cb2b2a1f3332ced97"
I20260924 10:39:44.981735 3969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbcaf14491694c21a40d7d745fc2f5db" candidate_uuid: "1b0aadae619d4689a5251f9ab93dce21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a574b056885428d87c58bc279a93d46"
I20260924 10:39:44.981846 4091 raft_consensus.cc:3037] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:44.981869 3969 raft_consensus.cc:3037] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:44.982900 3969 raft_consensus.cc:2454] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0aadae619d4689a5251f9ab93dce21 in term 1.
I20260924 10:39:44.982911 4091 raft_consensus.cc:2454] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0aadae619d4689a5251f9ab93dce21 in term 1.
I20260924 10:39:44.983173 4181 leader_election.cc:302] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0aadae619d4689a5251f9ab93dce21, 49e56cd06f5a464cb2b2a1f3332ced97; no voters:
I20260924 10:39:44.983397 4324 raft_consensus.cc:2781] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:44.983616 4324 raft_consensus.cc:686] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [term 1 LEADER]: Becoming Leader. State: Replica: 1b0aadae619d4689a5251f9ab93dce21, State: Running, Role: LEADER
I20260924 10:39:44.983911 4324 consensus_queue.cc:237] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:44.985076 4349 tablet_replica.cc:439] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21: TxnStatusTablet state changed. Reason: New leader 1b0aadae619d4689a5251f9ab93dce21. Latest consensus state: current_term: 1 leader_uuid: "1b0aadae619d4689a5251f9ab93dce21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } } }
I20260924 10:39:44.985194 4349 tablet_replica.cc:442] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:39:44.985445 4351 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 10:39:44.985600 4351 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 10:39:44.986279 3810 catalog_manager.cc:5495] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b0aadae619d4689a5251f9ab93dce21 (127.2.8.3). New cstate: current_term: 1 leader_uuid: "1b0aadae619d4689a5251f9ab93dce21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } health_report { overall_health: HEALTHY } } }
I20260924 10:39:45.043332 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:45.039151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.2.8.3:40493)
0924 10:39:45.039276 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:45.039280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:45.039311 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:39:45.039523 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:45.039525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:45.039580 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:45.039661 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:45.039990 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.040674 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.041923 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.042144 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.042230 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:45.042533 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:45.042976 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:45.043063 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:45.043160 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 10:39:45.043206 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:39:45.043373 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:45.039019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:40493 (local address 127.0.0.1:48786)
0924 10:39:45.039254 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:45.039269 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:45.039385 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:45.039693 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:45.039696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:45.039705 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:39:45.039809 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:45.039815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.040781 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.040785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:45.041791 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:45.041802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.042243 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.042257 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:45.042344 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:45.042401 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:39:45.043069 (+ 668us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:39:45.043082 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 10:39:45.043145 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 10:39:45.043227 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260924 10:39:45.048290 3969 raft_consensus.cc:1261] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46 [term 1 FOLLOWER]: Refusing update from remote peer 1b0aadae619d4689a5251f9ab93dce21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:45.048559 4091 raft_consensus.cc:1261] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 FOLLOWER]: Refusing update from remote peer 1b0aadae619d4689a5251f9ab93dce21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:45.049157 4324 consensus_queue.cc:1035] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:45.049332 4349 consensus_queue.cc:1035] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:45.052026 4325 tablet_replica.cc:439] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97: TxnStatusTablet state changed. Reason: New leader 1b0aadae619d4689a5251f9ab93dce21. Latest consensus state: current_term: 1 leader_uuid: "1b0aadae619d4689a5251f9ab93dce21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } } }
I20260924 10:39:45.052050 4326 tablet_replica.cc:439] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46: TxnStatusTablet state changed. Reason: New leader 1b0aadae619d4689a5251f9ab93dce21. Latest consensus state: current_term: 1 leader_uuid: "1b0aadae619d4689a5251f9ab93dce21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } } }
I20260924 10:39:45.052134 4325 tablet_replica.cc:442] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:39:45.052151 4326 tablet_replica.cc:442] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:39:45.055501 4324 tablet_replica.cc:439] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b0aadae619d4689a5251f9ab93dce21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } } }
I20260924 10:39:45.055619 4324 tablet_replica.cc:442] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:39:45.055730 4324 tablet_replica.cc:439] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b0aadae619d4689a5251f9ab93dce21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } } }
I20260924 10:39:45.055794 4324 tablet_replica.cc:442] T cbcaf14491694c21a40d7d745fc2f5db P 1b0aadae619d4689a5251f9ab93dce21: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:39:45.055869 4343 mvcc.cc:205] Tried to move back new op lower bound from 7332849193151868928 to 7332849192896233472. Current Snapshot: MvccSnapshot[applied={T|T < 7332849193151868928}]
I20260924 10:39:45.056538 4345 mvcc.cc:205] Tried to move back new op lower bound from 7332849193151868928 to 7332849192896233472. Current Snapshot: MvccSnapshot[applied={T|T < 7332849193151868928}]
I20260924 10:39:45.056804 4341 mvcc.cc:205] Tried to move back new op lower bound from 7332849193151868928 to 7332849192896233472. Current Snapshot: MvccSnapshot[applied={T|T < 7332849193151868928}]
I20260924 10:39:45.057003 4325 tablet_replica.cc:439] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b0aadae619d4689a5251f9ab93dce21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } } }
I20260924 10:39:45.057109 4325 tablet_replica.cc:442] T cbcaf14491694c21a40d7d745fc2f5db P 49e56cd06f5a464cb2b2a1f3332ced97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:39:45.056547 4326 tablet_replica.cc:439] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b0aadae619d4689a5251f9ab93dce21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } } }
I20260924 10:39:45.057283 4326 tablet_replica.cc:442] T cbcaf14491694c21a40d7d745fc2f5db P 8a574b056885428d87c58bc279a93d46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:39:45.068195 4348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:45.064009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51018 (local address 127.2.8.2:34367)
0924 10:39:45.064142 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:45.064146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:45.064363 (+ 217us) server_negotiation.cc:407] Connection header received
0924 10:39:45.064466 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:45.064468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:45.064517 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:45.064598 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:45.064858 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.065437 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.066682 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.067020 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.067128 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:45.067385 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:45.067783 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:45.067870 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:45.067969 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 10:39:45.068017 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:39:45.078940 4364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:45.075425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:40493 (local address 127.2.8.2:58269)
0924 10:39:45.075799 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:45.075810 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:45.075949 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:45.076190 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:45.076192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:45.076200 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:39:45.076296 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:45.076302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.077036 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.077040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:45.077653 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:45.077661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.078154 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.078173 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:45.078257 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:45.078352 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:39:45.078673 (+ 321us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:39:45.078679 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:39:45.078738 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 10:39:45.078816 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":115,"threads_started":1}
I20260924 10:39:45.078944 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:45.075510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:58269 (local address 127.2.8.3:40493)
0924 10:39:45.075636 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:45.075641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:45.075902 (+ 261us) server_negotiation.cc:407] Connection header received
0924 10:39:45.076057 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:45.076059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:45.076101 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:45.076181 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:45.076415 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.076895 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.077789 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.078046 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.078169 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:45.078512 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:45.078583 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:45.078659 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:45.078748 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 10:39:45.078792 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:39:45.106879 4348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:45.102845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:57741 (local address 127.2.8.2:34367)
0924 10:39:45.102953 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:45.102957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:45.103237 (+ 280us) server_negotiation.cc:407] Connection header received
0924 10:39:45.103343 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:45.103345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:45.103384 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:45.103444 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:45.103738 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.104233 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.105603 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.105954 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.106091 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:45.106440 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:45.106558 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:45.106629 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:45.106738 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 10:39:45.106778 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:39:45.107158 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:45.102560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:34367 (local address 127.2.8.3:57741)
0924 10:39:45.103138 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:45.103155 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:45.103262 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:45.103478 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:45.103480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:45.103488 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:39:45.103613 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:45.103619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.104344 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.104347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:45.105412 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:45.105427 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.106100 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.106131 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:45.106221 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:45.106267 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:39:45.106639 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:39:45.106646 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:39:45.106724 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 10:39:45.106800 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":105,"threads_started":1}
I20260924 10:39:45.130836 3812 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:39:45.131085 3812 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:39:45.133725 3812 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9: Sending DeleteTablet for 3 replicas of tablet e3735b746dd6443a96d229bcff064ef7
I20260924 10:39:45.134394 4089 tablet_service.cc:1514] Processing DeleteTablet for tablet e3735b746dd6443a96d229bcff064ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:45 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:51004
I20260924 10:39:45.134718 3949 tablet_service.cc:1514] Processing DeleteTablet for tablet e3735b746dd6443a96d229bcff064ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:45 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:39914
I20260924 10:39:45.135075 4229 tablet_service.cc:1514] Processing DeleteTablet for tablet e3735b746dd6443a96d229bcff064ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:45 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:48772
I20260924 10:39:45.135634 4371 tablet_replica.cc:330] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21: stopping tablet replica
I20260924 10:39:45.135833 3812 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:39:45.136204 3812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:39:45.136345 4371 raft_consensus.cc:2229] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:39:45.136610 4369 tablet_replica.cc:330] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97: stopping tablet replica
I20260924 10:39:45.136868 4371 raft_consensus.cc:2258] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:45.137213 4369 raft_consensus.cc:2229] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:39:45.138672 4371 ts_tablet_manager.cc:1905] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:45.139094 4369 raft_consensus.cc:2258] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:45.140275 4089 tablet_service.cc:1467] Processing CreateTablet for tablet 9db81f92beb24992a45e6a987d813d2c (DEFAULT_TABLE table=test-table [id=f12abefe613c4f63a639e8b5698c41ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:45.140468 4370 tablet_replica.cc:330] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46: stopping tablet replica
I20260924 10:39:45.140545 4089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db81f92beb24992a45e6a987d813d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:45.140858 4370 raft_consensus.cc:2229] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:39:45.141520 4229 tablet_service.cc:1467] Processing CreateTablet for tablet 9db81f92beb24992a45e6a987d813d2c (DEFAULT_TABLE table=test-table [id=f12abefe613c4f63a639e8b5698c41ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:45.141776 4229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db81f92beb24992a45e6a987d813d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:45.141626 3949 tablet_service.cc:1467] Processing CreateTablet for tablet 9db81f92beb24992a45e6a987d813d2c (DEFAULT_TABLE table=test-table [id=f12abefe613c4f63a639e8b5698c41ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:45.142557 3949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db81f92beb24992a45e6a987d813d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:45.143713 4318 tablet_bootstrap.cc:492] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97: Bootstrap starting.
I20260924 10:39:45.144560 4318 tablet_bootstrap.cc:654] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:45.144968 4319 tablet_bootstrap.cc:492] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46: Bootstrap starting.
I20260924 10:39:45.145798 4369 ts_tablet_manager.cc:1905] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:45.145294 4370 raft_consensus.cc:2258] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:45.146275 4319 tablet_bootstrap.cc:654] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:45.147680 4319 tablet_bootstrap.cc:492] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46: No bootstrap required, opened a new log
I20260924 10:39:45.147787 4319 ts_tablet_manager.cc:1397] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:39:45.148644 4319 raft_consensus.cc:348] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:45.148766 4319 raft_consensus.cc:374] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:45.148810 4319 raft_consensus.cc:729] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a574b056885428d87c58bc279a93d46, State: Initialized, Role: FOLLOWER
I20260924 10:39:45.148968 4319 consensus_queue.cc:260] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:45.149318 4320 tablet_bootstrap.cc:492] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21: Bootstrap starting.
I20260924 10:39:45.149974 4319 ts_tablet_manager.cc:1428] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:39:45.150337 4320 tablet_bootstrap.cc:654] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:45.151432 4318 tablet_bootstrap.cc:492] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97: No bootstrap required, opened a new log
I20260924 10:39:45.151525 4318 ts_tablet_manager.cc:1397] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20260924 10:39:45.152025 4318 raft_consensus.cc:348] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:45.152141 4318 raft_consensus.cc:374] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:45.152179 4318 raft_consensus.cc:729] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e56cd06f5a464cb2b2a1f3332ced97, State: Initialized, Role: FOLLOWER
I20260924 10:39:45.152331 4318 consensus_queue.cc:260] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:45.152639 4318 ts_tablet_manager.cc:1428] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:39:45.153875 4320 tablet_bootstrap.cc:492] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21: No bootstrap required, opened a new log
I20260924 10:39:45.153976 4320 ts_tablet_manager.cc:1397] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260924 10:39:45.154088 4370 ts_tablet_manager.cc:1905] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:45.154448 4320 raft_consensus.cc:348] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:45.154583 4320 raft_consensus.cc:374] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:45.154618 4320 raft_consensus.cc:729] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0aadae619d4689a5251f9ab93dce21, State: Initialized, Role: FOLLOWER
I20260924 10:39:45.154747 4320 consensus_queue.cc:260] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:45.155579 4320 ts_tablet_manager.cc:1428] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 10:39:45.158376 4370 ts_tablet_manager.cc:1918] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:39:45.159945 4369 ts_tablet_manager.cc:1918] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:39:45.160055 4371 ts_tablet_manager.cc:1918] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:39:45.160110 4369 log.cc:1196] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/e3735b746dd6443a96d229bcff064ef7
I20260924 10:39:45.160158 4371 log.cc:1196] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/e3735b746dd6443a96d229bcff064ef7
I20260924 10:39:45.160478 4371 ts_tablet_manager.cc:1939] T e3735b746dd6443a96d229bcff064ef7 P 1b0aadae619d4689a5251f9ab93dce21: Deleting consensus metadata
I20260924 10:39:45.160696 4369 ts_tablet_manager.cc:1939] T e3735b746dd6443a96d229bcff064ef7 P 49e56cd06f5a464cb2b2a1f3332ced97: Deleting consensus metadata
I20260924 10:39:45.161410 4370 log.cc:1196] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/wal/wals/e3735b746dd6443a96d229bcff064ef7
I20260924 10:39:45.161753 4370 ts_tablet_manager.cc:1939] T e3735b746dd6443a96d229bcff064ef7 P 8a574b056885428d87c58bc279a93d46: Deleting consensus metadata
I20260924 10:39:45.161875 3797 catalog_manager.cc:4841] TS 49e56cd06f5a464cb2b2a1f3332ced97 (127.2.8.2:34367): tablet e3735b746dd6443a96d229bcff064ef7 (table test-table [id=b804c8f2cf7047c4a8675ab8cb78312b]) successfully deleted
I20260924 10:39:45.162475 3798 catalog_manager.cc:4841] TS 1b0aadae619d4689a5251f9ab93dce21 (127.2.8.3:40493): tablet e3735b746dd6443a96d229bcff064ef7 (table test-table [id=b804c8f2cf7047c4a8675ab8cb78312b]) successfully deleted
I20260924 10:39:45.163540 3798 catalog_manager.cc:4841] TS 8a574b056885428d87c58bc279a93d46 (127.2.8.1:45995): tablet e3735b746dd6443a96d229bcff064ef7 (table test-table [id=b804c8f2cf7047c4a8675ab8cb78312b]) successfully deleted
I20260924 10:39:45.417660 4324 raft_consensus.cc:482] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:39:45.417806 4324 raft_consensus.cc:504] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:45.418186 4324 leader_election.cc:288] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a574b056885428d87c58bc279a93d46 (127.2.8.1:45995), 49e56cd06f5a464cb2b2a1f3332ced97 (127.2.8.2:34367)
I20260924 10:39:45.418681 4091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db81f92beb24992a45e6a987d813d2c" candidate_uuid: "1b0aadae619d4689a5251f9ab93dce21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" is_pre_election: true
I20260924 10:39:45.418699 3969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db81f92beb24992a45e6a987d813d2c" candidate_uuid: "1b0aadae619d4689a5251f9ab93dce21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a574b056885428d87c58bc279a93d46" is_pre_election: true
I20260924 10:39:45.418816 4091 raft_consensus.cc:2454] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b0aadae619d4689a5251f9ab93dce21 in term 0.
I20260924 10:39:45.418855 3969 raft_consensus.cc:2454] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b0aadae619d4689a5251f9ab93dce21 in term 0.
I20260924 10:39:45.419174 4181 leader_election.cc:302] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0aadae619d4689a5251f9ab93dce21, 49e56cd06f5a464cb2b2a1f3332ced97; no voters:
I20260924 10:39:45.419378 4324 raft_consensus.cc:2781] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:39:45.419430 4324 raft_consensus.cc:482] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:39:45.419457 4324 raft_consensus.cc:3037] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:45.420469 4324 raft_consensus.cc:504] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:45.420874 4324 leader_election.cc:288] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [CANDIDATE]: Term 1 election: Requested vote from peers 8a574b056885428d87c58bc279a93d46 (127.2.8.1:45995), 49e56cd06f5a464cb2b2a1f3332ced97 (127.2.8.2:34367)
I20260924 10:39:45.421303 4091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db81f92beb24992a45e6a987d813d2c" candidate_uuid: "1b0aadae619d4689a5251f9ab93dce21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e56cd06f5a464cb2b2a1f3332ced97"
I20260924 10:39:45.421346 3969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db81f92beb24992a45e6a987d813d2c" candidate_uuid: "1b0aadae619d4689a5251f9ab93dce21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a574b056885428d87c58bc279a93d46"
I20260924 10:39:45.421396 4091 raft_consensus.cc:3037] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:45.421494 3969 raft_consensus.cc:3037] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:45.422495 4091 raft_consensus.cc:2454] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0aadae619d4689a5251f9ab93dce21 in term 1.
I20260924 10:39:45.422598 3969 raft_consensus.cc:2454] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b0aadae619d4689a5251f9ab93dce21 in term 1.
I20260924 10:39:45.422832 4181 leader_election.cc:302] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b0aadae619d4689a5251f9ab93dce21, 49e56cd06f5a464cb2b2a1f3332ced97; no voters:
I20260924 10:39:45.423010 4324 raft_consensus.cc:2781] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:45.423103 4324 raft_consensus.cc:686] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [term 1 LEADER]: Becoming Leader. State: Replica: 1b0aadae619d4689a5251f9ab93dce21, State: Running, Role: LEADER
I20260924 10:39:45.423249 4324 consensus_queue.cc:237] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:45.424825 3810 catalog_manager.cc:5495] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b0aadae619d4689a5251f9ab93dce21 (127.2.8.3). New cstate: current_term: 1 leader_uuid: "1b0aadae619d4689a5251f9ab93dce21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } health_report { overall_health: HEALTHY } } }
I20260924 10:39:45.451887 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:45.448487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.2.8.3:40493)
0924 10:39:45.448625 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:45.448632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:45.448722 (+ 90us) server_negotiation.cc:407] Connection header received
0924 10:39:45.448879 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:45.448882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:45.448942 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:45.449068 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:45.449364 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.449888 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.450758 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.450995 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.451114 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:45.451396 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:45.451477 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:45.451558 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:45.451667 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 10:39:45.451726 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:39:45.460493 4366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:45.457511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:40493 (local address 127.2.8.3:38611)
0924 10:39:45.457637 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:45.457651 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:45.457729 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:45.457975 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:45.457978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:45.457984 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:39:45.458111 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:45.458123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.458823 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.458826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:45.459425 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:45.459433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.459841 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.459855 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:45.459930 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:45.459958 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:39:45.460220 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:39:45.460226 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:39:45.460290 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 10:39:45.460388 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":55}
I20260924 10:39:45.460493 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:45.457571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:38611 (local address 127.2.8.3:40493)
0924 10:39:45.457677 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:45.457684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:45.457699 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:39:45.457831 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:45.457834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:45.457894 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:45.457970 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:45.458224 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.458713 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.459538 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:45.459752 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:45.459864 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:45.460076 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:45.460143 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:45.460219 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:45.460298 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 10:39:45.460350 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":46}
I20260924 10:39:45.465719 3969 raft_consensus.cc:1261] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46 [term 1 FOLLOWER]: Refusing update from remote peer 1b0aadae619d4689a5251f9ab93dce21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:45.465755 4091 raft_consensus.cc:1261] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 FOLLOWER]: Refusing update from remote peer 1b0aadae619d4689a5251f9ab93dce21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:45.466154 4356 consensus_queue.cc:1035] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:45.466293 4349 consensus_queue.cc:1035] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:39:45.479112 4343 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 10:39:47.034600 3810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:39:47.034789 3810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:39:47.037200 3810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9: Sending DeleteTablet for 3 replicas of tablet 9db81f92beb24992a45e6a987d813d2c
I20260924 10:39:47.037892 4229 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db81f92beb24992a45e6a987d813d2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:47 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:48772
I20260924 10:39:47.037920 3949 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db81f92beb24992a45e6a987d813d2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:47 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:39914
I20260924 10:39:47.038015 4089 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db81f92beb24992a45e6a987d813d2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:47 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:51004
I20260924 10:39:47.038247 4400 tablet_replica.cc:330] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21: stopping tablet replica
I20260924 10:39:47.038424 4400 raft_consensus.cc:2229] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:39:47.038437 4402 tablet_replica.cc:330] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97: stopping tablet replica
I20260924 10:39:47.038556 4402 raft_consensus.cc:2229] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:39:47.038725 4402 raft_consensus.cc:2258] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:47.038821 4400 raft_consensus.cc:2258] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:47.039060 4401 tablet_replica.cc:330] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46: stopping tablet replica
I20260924 10:39:47.039197 4401 raft_consensus.cc:2229] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:39:47.039314 4401 raft_consensus.cc:2258] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:47.039443 4402 ts_tablet_manager.cc:1905] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:47.039830 4400 ts_tablet_manager.cc:1905] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:47.042371 4402 ts_tablet_manager.cc:1918] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:39:47.042462 4402 log.cc:1196] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/9db81f92beb24992a45e6a987d813d2c
I20260924 10:39:47.042753 4402 ts_tablet_manager.cc:1939] T 9db81f92beb24992a45e6a987d813d2c P 49e56cd06f5a464cb2b2a1f3332ced97: Deleting consensus metadata
I20260924 10:39:47.042758 4400 ts_tablet_manager.cc:1918] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:39:47.042824 4400 log.cc:1196] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/9db81f92beb24992a45e6a987d813d2c
I20260924 10:39:47.043147 4400 ts_tablet_manager.cc:1939] T 9db81f92beb24992a45e6a987d813d2c P 1b0aadae619d4689a5251f9ab93dce21: Deleting consensus metadata
I20260924 10:39:47.043624 3797 catalog_manager.cc:4841] TS 49e56cd06f5a464cb2b2a1f3332ced97 (127.2.8.2:34367): tablet 9db81f92beb24992a45e6a987d813d2c (table test-table [id=f12abefe613c4f63a639e8b5698c41ef]) successfully deleted
I20260924 10:39:47.043875 3798 catalog_manager.cc:4841] TS 1b0aadae619d4689a5251f9ab93dce21 (127.2.8.3:40493): tablet 9db81f92beb24992a45e6a987d813d2c (table test-table [id=f12abefe613c4f63a639e8b5698c41ef]) successfully deleted
I20260924 10:39:47.044363 4401 ts_tablet_manager.cc:1905] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:47.047164 4401 ts_tablet_manager.cc:1918] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:39:47.047271 4401 log.cc:1196] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/wal/wals/9db81f92beb24992a45e6a987d813d2c
I20260924 10:39:47.047519 4401 ts_tablet_manager.cc:1939] T 9db81f92beb24992a45e6a987d813d2c P 8a574b056885428d87c58bc279a93d46: Deleting consensus metadata
I20260924 10:39:47.048182 3798 catalog_manager.cc:4841] TS 8a574b056885428d87c58bc279a93d46 (127.2.8.1:45995): tablet 9db81f92beb24992a45e6a987d813d2c (table test-table [id=f12abefe613c4f63a639e8b5698c41ef]) successfully deleted
Sep 24 10:39:47 dist-test-slave-j38h krb5kdc[3767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246387, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:39:47 dist-test-slave-j38h krb5kdc[3767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246387, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:47.076500 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:47.059662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41984 (local address 127.2.8.62:43213)
0924 10:39:47.059962 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:47.059985 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:47.060001 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:39:47.060047 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:47.060049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:47.060092 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:47.060176 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:47.060879 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.061645 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.062548 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.062849 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.062940 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:47.072868 (+ 9928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:47.072873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:47.072875 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:47.072907 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:47.074771 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:47.075204 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:47.075207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:47.075208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:47.075341 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:47.075594 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:47.075596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:47.075598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:47.075799 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:47.075908 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:47.076096 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 10:39:47.076211 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":100,"threads_started":1}
I20260924 10:39:47.079566 3810 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:39:47.080036 3810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:39:47.084882 4089 tablet_service.cc:1467] Processing CreateTablet for tablet add447a5d6c74654a4f66bc239dca4df (DEFAULT_TABLE table=test-table [id=b5cbfc2eb2d94562972658550c1fc2fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:47.084913 4229 tablet_service.cc:1467] Processing CreateTablet for tablet add447a5d6c74654a4f66bc239dca4df (DEFAULT_TABLE table=test-table [id=b5cbfc2eb2d94562972658550c1fc2fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:47.084880 3949 tablet_service.cc:1467] Processing CreateTablet for tablet add447a5d6c74654a4f66bc239dca4df (DEFAULT_TABLE table=test-table [id=b5cbfc2eb2d94562972658550c1fc2fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:47.085201 4229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add447a5d6c74654a4f66bc239dca4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:47.085201 4089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add447a5d6c74654a4f66bc239dca4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:47.085237 3949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add447a5d6c74654a4f66bc239dca4df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:47.087702 4411 tablet_bootstrap.cc:492] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46: Bootstrap starting.
I20260924 10:39:47.087868 4412 tablet_bootstrap.cc:492] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97: Bootstrap starting.
I20260924 10:39:47.088430 4413 tablet_bootstrap.cc:492] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21: Bootstrap starting.
I20260924 10:39:47.088763 4411 tablet_bootstrap.cc:654] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:47.088799 4412 tablet_bootstrap.cc:654] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:47.089383 4413 tablet_bootstrap.cc:654] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:47.090128 4412 tablet_bootstrap.cc:492] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97: No bootstrap required, opened a new log
I20260924 10:39:47.090164 4411 tablet_bootstrap.cc:492] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46: No bootstrap required, opened a new log
I20260924 10:39:47.090229 4412 ts_tablet_manager.cc:1397] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:47.090247 4411 ts_tablet_manager.cc:1397] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 10:39:47.090547 4413 tablet_bootstrap.cc:492] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21: No bootstrap required, opened a new log
I20260924 10:39:47.090636 4413 ts_tablet_manager.cc:1397] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:47.090723 4412 raft_consensus.cc:348] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.090858 4412 raft_consensus.cc:374] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:47.090888 4412 raft_consensus.cc:729] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e56cd06f5a464cb2b2a1f3332ced97, State: Initialized, Role: FOLLOWER
I20260924 10:39:47.091034 4412 consensus_queue.cc:260] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.091048 4413 raft_consensus.cc:348] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.091152 4413 raft_consensus.cc:374] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:47.091192 4413 raft_consensus.cc:729] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0aadae619d4689a5251f9ab93dce21, State: Initialized, Role: FOLLOWER
I20260924 10:39:47.091162 4411 raft_consensus.cc:348] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.091266 4411 raft_consensus.cc:374] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:47.091320 4411 raft_consensus.cc:729] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a574b056885428d87c58bc279a93d46, State: Initialized, Role: FOLLOWER
I20260924 10:39:47.091328 4413 consensus_queue.cc:260] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.091435 4412 ts_tablet_manager.cc:1428] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:39:47.091437 4411 consensus_queue.cc:260] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.091663 4413 ts_tablet_manager.cc:1428] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:39:47.091889 4411 ts_tablet_manager.cc:1428] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 10:39:47.268337 4418 raft_consensus.cc:482] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:39:47.268501 4418 raft_consensus.cc:504] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.269881 4418 leader_election.cc:288] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e56cd06f5a464cb2b2a1f3332ced97 (127.2.8.2:34367), 1b0aadae619d4689a5251f9ab93dce21 (127.2.8.3:40493)
I20260924 10:39:47.274024 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:47.270225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:57149 (local address 127.2.8.2:34367)
0924 10:39:47.270564 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:47.270573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:47.270647 (+ 74us) server_negotiation.cc:407] Connection header received
0924 10:39:47.270821 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:47.270824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:47.270875 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:47.270967 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:47.271269 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.271922 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.273016 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.273419 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.273543 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:47.273614 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:47.273815 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 10:39:47.273889 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260924 10:39:47.274034 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:47.270124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:34367 (local address 127.2.8.1:57149)
0924 10:39:47.270543 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:47.270560 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:47.270711 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:47.270989 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:47.270992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:47.271008 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:47.271159 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:47.271164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.272050 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.272053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:47.272865 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:47.272874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.273582 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.273586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:47.273691 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:47.273709 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 10:39:47.273799 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 10:39:47.273877 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":125,"threads_started":1}
I20260924 10:39:47.274585 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:47.270172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:40493 (local address 127.2.8.1:35083)
0924 10:39:47.270650 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:47.270669 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:47.270832 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:47.271153 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:47.271156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:47.271175 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:47.271332 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:47.271339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.272313 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.272320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:47.273282 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:47.273292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.274028 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.274032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:47.274150 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:47.274186 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 10:39:47.274309 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 10:39:47.274409 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":129,"threads_started":1}
I20260924 10:39:47.274595 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:47.270333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:35083 (local address 127.2.8.3:40493)
0924 10:39:47.270618 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:47.270629 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:47.270779 (+ 150us) server_negotiation.cc:407] Connection header received
0924 10:39:47.270967 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:47.270971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:47.271031 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:47.271144 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:47.271493 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.272181 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.273427 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.273889 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.274029 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:47.274105 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:47.274346 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 10:39:47.274474 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":116,"threads_started":1}
I20260924 10:39:47.274788 4091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add447a5d6c74654a4f66bc239dca4df" candidate_uuid: "8a574b056885428d87c58bc279a93d46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" is_pre_election: true
I20260924 10:39:47.275015 4091 raft_consensus.cc:2454] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a574b056885428d87c58bc279a93d46 in term 0.
I20260924 10:39:47.275072 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add447a5d6c74654a4f66bc239dca4df" candidate_uuid: "8a574b056885428d87c58bc279a93d46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0aadae619d4689a5251f9ab93dce21" is_pre_election: true
I20260924 10:39:47.275192 4249 raft_consensus.cc:2454] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a574b056885428d87c58bc279a93d46 in term 0.
I20260924 10:39:47.275560 3901 leader_election.cc:302] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b0aadae619d4689a5251f9ab93dce21, 49e56cd06f5a464cb2b2a1f3332ced97, 8a574b056885428d87c58bc279a93d46; no voters:
I20260924 10:39:47.275884 4418 raft_consensus.cc:2781] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:39:47.275957 4418 raft_consensus.cc:482] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:39:47.276028 4418 raft_consensus.cc:3037] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:47.277268 4418 raft_consensus.cc:504] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.277637 4418 leader_election.cc:288] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [CANDIDATE]: Term 1 election: Requested vote from peers 49e56cd06f5a464cb2b2a1f3332ced97 (127.2.8.2:34367), 1b0aadae619d4689a5251f9ab93dce21 (127.2.8.3:40493)
I20260924 10:39:47.278052 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add447a5d6c74654a4f66bc239dca4df" candidate_uuid: "8a574b056885428d87c58bc279a93d46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0aadae619d4689a5251f9ab93dce21"
I20260924 10:39:47.278033 4091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add447a5d6c74654a4f66bc239dca4df" candidate_uuid: "8a574b056885428d87c58bc279a93d46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e56cd06f5a464cb2b2a1f3332ced97"
I20260924 10:39:47.278173 4249 raft_consensus.cc:3037] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:47.278173 4091 raft_consensus.cc:3037] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:47.279297 4249 raft_consensus.cc:2454] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a574b056885428d87c58bc279a93d46 in term 1.
I20260924 10:39:47.279297 4091 raft_consensus.cc:2454] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a574b056885428d87c58bc279a93d46 in term 1.
I20260924 10:39:47.279623 3902 leader_election.cc:302] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b0aadae619d4689a5251f9ab93dce21, 49e56cd06f5a464cb2b2a1f3332ced97, 8a574b056885428d87c58bc279a93d46; no voters:
I20260924 10:39:47.279798 4418 raft_consensus.cc:2781] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:47.280030 4418 raft_consensus.cc:686] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [term 1 LEADER]: Becoming Leader. State: Replica: 8a574b056885428d87c58bc279a93d46, State: Running, Role: LEADER
I20260924 10:39:47.280297 4418 consensus_queue.cc:237] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.282482 3812 catalog_manager.cc:5495] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a574b056885428d87c58bc279a93d46 (127.2.8.1). New cstate: current_term: 1 leader_uuid: "8a574b056885428d87c58bc279a93d46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } health_report { overall_health: UNKNOWN } } }
I20260924 10:39:47.336091 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:47.332315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39930 (local address 127.2.8.1:45995)
0924 10:39:47.332583 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:47.332589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:47.332601 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:39:47.332671 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:47.332673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:47.332721 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:47.332827 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:47.333088 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.333634 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.334522 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.334755 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.334881 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:47.335079 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:47.335609 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:47.335709 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:47.335862 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 10:39:47.335938 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":123,"threads_started":1}
I20260924 10:39:47.346772 4429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:47.343236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:40493 (local address 127.2.8.1:44349)
0924 10:39:47.343602 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:47.343616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:47.343705 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:47.343981 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:47.343983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:47.343989 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:39:47.344082 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:47.344087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.344759 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.344762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:47.345585 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:47.345595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.346009 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.346023 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:47.346094 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:47.346136 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:39:47.346511 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:39:47.346516 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:39:47.346572 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 10:39:47.346636 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":117,"threads_started":1}
I20260924 10:39:47.346807 4422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:47.343481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:44349 (local address 127.2.8.3:40493)
0924 10:39:47.343617 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:47.343621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:47.343665 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:39:47.343829 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:47.343832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:47.343888 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:47.343952 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:47.344176 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.344657 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.345692 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.345901 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.345976 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:47.346314 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:47.346421 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:47.346490 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:47.346640 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 10:39:47.346692 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:39:47.352643 4249 raft_consensus.cc:1261] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21 [term 1 FOLLOWER]: Refusing update from remote peer 8a574b056885428d87c58bc279a93d46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:47.352643 4091 raft_consensus.cc:1261] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 FOLLOWER]: Refusing update from remote peer 8a574b056885428d87c58bc279a93d46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:47.353190 4423 consensus_queue.cc:1035] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:47.353350 4418 consensus_queue.cc:1035] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:47.370929 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:47.367417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:57215 (local address 127.2.8.1:45995)
0924 10:39:47.367549 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:47.367553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:47.367667 (+ 114us) server_negotiation.cc:407] Connection header received
0924 10:39:47.367807 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:47.367813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:47.367883 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:47.367966 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:47.368245 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.368818 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.369808 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.370065 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.370188 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:47.370457 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:47.370538 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:47.370627 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:47.370721 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 10:39:47.370773 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:39:47.370939 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:47.367366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:45995 (local address 127.2.8.3:57215)
0924 10:39:47.367594 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:47.367609 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:47.367706 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:47.367994 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:47.367997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:47.368006 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:39:47.368116 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:47.368125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.368945 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.368952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:47.369675 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:47.369684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.370172 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.370188 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:47.370281 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:47.370318 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:39:47.370630 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:39:47.370639 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 10:39:47.370710 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 10:39:47.370801 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20260924 10:39:47.393826 3812 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:39:47.394033 3812 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:39:47.396131 3812 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9: Sending DeleteTablet for 3 replicas of tablet add447a5d6c74654a4f66bc239dca4df
I20260924 10:39:47.396700 3949 tablet_service.cc:1514] Processing DeleteTablet for tablet add447a5d6c74654a4f66bc239dca4df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:47 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:39914
I20260924 10:39:47.396795 4089 tablet_service.cc:1514] Processing DeleteTablet for tablet add447a5d6c74654a4f66bc239dca4df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:47 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:51004
I20260924 10:39:47.396896 4401 tablet_replica.cc:330] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46: stopping tablet replica
I20260924 10:39:47.397007 4402 tablet_replica.cc:330] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97: stopping tablet replica
I20260924 10:39:47.397027 4401 raft_consensus.cc:2229] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:39:47.397120 4402 raft_consensus.cc:2229] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:39:47.397234 4402 raft_consensus.cc:2258] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:47.397399 4401 raft_consensus.cc:2258] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:47.397508 3812 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:39:47.397948 3812 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:39:47.397985 4402 ts_tablet_manager.cc:1905] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:47.398612 4401 ts_tablet_manager.cc:1905] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:47.400580 4402 ts_tablet_manager.cc:1918] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:39:47.400689 4402 log.cc:1196] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/add447a5d6c74654a4f66bc239dca4df
I20260924 10:39:47.401002 4402 ts_tablet_manager.cc:1939] T add447a5d6c74654a4f66bc239dca4df P 49e56cd06f5a464cb2b2a1f3332ced97: Deleting consensus metadata
I20260924 10:39:47.401262 4401 ts_tablet_manager.cc:1918] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:39:47.401672 3797 catalog_manager.cc:4841] TS 49e56cd06f5a464cb2b2a1f3332ced97 (127.2.8.2:34367): tablet add447a5d6c74654a4f66bc239dca4df (table test-table [id=b5cbfc2eb2d94562972658550c1fc2fd]) successfully deleted
I20260924 10:39:47.402263 4229 tablet_service.cc:1514] Processing DeleteTablet for tablet add447a5d6c74654a4f66bc239dca4df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:47 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:48772
I20260924 10:39:47.402472 4400 tablet_replica.cc:330] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21: stopping tablet replica
I20260924 10:39:47.402616 4400 raft_consensus.cc:2229] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:39:47.402746 4400 raft_consensus.cc:2258] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:47.403491 4089 tablet_service.cc:1467] Processing CreateTablet for tablet 95ffe2cfbcbf4b9c811f0faa246b49b3 (DEFAULT_TABLE table=test-table [id=9d5bd0583e05412b8a8e5a609e31c4fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:47.403605 4400 ts_tablet_manager.cc:1905] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:47.403721 4089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95ffe2cfbcbf4b9c811f0faa246b49b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:47.403669 4229 tablet_service.cc:1467] Processing CreateTablet for tablet 95ffe2cfbcbf4b9c811f0faa246b49b3 (DEFAULT_TABLE table=test-table [id=9d5bd0583e05412b8a8e5a609e31c4fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:47.403949 4229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95ffe2cfbcbf4b9c811f0faa246b49b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:47.403381 3949 tablet_service.cc:1467] Processing CreateTablet for tablet 95ffe2cfbcbf4b9c811f0faa246b49b3 (DEFAULT_TABLE table=test-table [id=9d5bd0583e05412b8a8e5a609e31c4fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:47.406181 4412 tablet_bootstrap.cc:492] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97: Bootstrap starting.
I20260924 10:39:47.407262 3949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95ffe2cfbcbf4b9c811f0faa246b49b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:47.407362 4412 tablet_bootstrap.cc:654] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:47.408646 4400 ts_tablet_manager.cc:1918] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:39:47.408737 4400 log.cc:1196] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/add447a5d6c74654a4f66bc239dca4df
I20260924 10:39:47.409238 4412 tablet_bootstrap.cc:492] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97: No bootstrap required, opened a new log
I20260924 10:39:47.409332 4412 ts_tablet_manager.cc:1397] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260924 10:39:47.409713 4411 tablet_bootstrap.cc:492] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 8a574b056885428d87c58bc279a93d46: Bootstrap starting.
I20260924 10:39:47.409789 4412 raft_consensus.cc:348] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.409919 4412 raft_consensus.cc:374] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:47.409960 4412 raft_consensus.cc:729] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e56cd06f5a464cb2b2a1f3332ced97, State: Initialized, Role: FOLLOWER
I20260924 10:39:47.409970 4401 log.cc:1196] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-0/wal/wals/add447a5d6c74654a4f66bc239dca4df
I20260924 10:39:47.410104 4412 consensus_queue.cc:260] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.410280 4401 ts_tablet_manager.cc:1939] T add447a5d6c74654a4f66bc239dca4df P 8a574b056885428d87c58bc279a93d46: Deleting consensus metadata
I20260924 10:39:47.410601 4411 tablet_bootstrap.cc:654] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 8a574b056885428d87c58bc279a93d46: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:47.410970 3798 catalog_manager.cc:4841] TS 8a574b056885428d87c58bc279a93d46 (127.2.8.1:45995): tablet add447a5d6c74654a4f66bc239dca4df (table test-table [id=b5cbfc2eb2d94562972658550c1fc2fd]) successfully deleted
I20260924 10:39:47.411386 4412 ts_tablet_manager.cc:1428] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 10:39:47.411834 4413 tablet_bootstrap.cc:492] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21: Bootstrap starting.
I20260924 10:39:47.411834 4400 ts_tablet_manager.cc:1939] T add447a5d6c74654a4f66bc239dca4df P 1b0aadae619d4689a5251f9ab93dce21: Deleting consensus metadata
I20260924 10:39:47.412727 4413 tablet_bootstrap.cc:654] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:47.413266 3798 catalog_manager.cc:4841] TS 1b0aadae619d4689a5251f9ab93dce21 (127.2.8.3:40493): tablet add447a5d6c74654a4f66bc239dca4df (table test-table [id=b5cbfc2eb2d94562972658550c1fc2fd]) successfully deleted
I20260924 10:39:47.414008 4413 tablet_bootstrap.cc:492] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21: No bootstrap required, opened a new log
I20260924 10:39:47.414110 4413 ts_tablet_manager.cc:1397] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 10:39:47.414506 4411 tablet_bootstrap.cc:492] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 8a574b056885428d87c58bc279a93d46: No bootstrap required, opened a new log
I20260924 10:39:47.414592 4411 ts_tablet_manager.cc:1397] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 8a574b056885428d87c58bc279a93d46: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260924 10:39:47.414491 4413 raft_consensus.cc:348] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.414638 4413 raft_consensus.cc:374] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:47.414690 4413 raft_consensus.cc:729] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b0aadae619d4689a5251f9ab93dce21, State: Initialized, Role: FOLLOWER
I20260924 10:39:47.414852 4413 consensus_queue.cc:260] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.414997 4411 raft_consensus.cc:348] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.415104 4411 raft_consensus.cc:374] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:47.415133 4411 raft_consensus.cc:729] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a574b056885428d87c58bc279a93d46, State: Initialized, Role: FOLLOWER
I20260924 10:39:47.415205 4413 ts_tablet_manager.cc:1428] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:39:47.415264 4411 consensus_queue.cc:260] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 8a574b056885428d87c58bc279a93d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.415488 4411 ts_tablet_manager.cc:1428] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 8a574b056885428d87c58bc279a93d46: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:39:47.480841 4417 raft_consensus.cc:482] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:39:47.481021 4417 raft_consensus.cc:504] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.481570 4417 leader_election.cc:288] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a574b056885428d87c58bc279a93d46 (127.2.8.1:45995), 1b0aadae619d4689a5251f9ab93dce21 (127.2.8.3:40493)
I20260924 10:39:47.482055 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ffe2cfbcbf4b9c811f0faa246b49b3" candidate_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0aadae619d4689a5251f9ab93dce21" is_pre_election: true
I20260924 10:39:47.482060 3969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ffe2cfbcbf4b9c811f0faa246b49b3" candidate_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a574b056885428d87c58bc279a93d46" is_pre_election: true
I20260924 10:39:47.482206 3969 raft_consensus.cc:2454] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e56cd06f5a464cb2b2a1f3332ced97 in term 0.
I20260924 10:39:47.482214 4249 raft_consensus.cc:2454] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e56cd06f5a464cb2b2a1f3332ced97 in term 0.
I20260924 10:39:47.482535 4042 leader_election.cc:302] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e56cd06f5a464cb2b2a1f3332ced97, 8a574b056885428d87c58bc279a93d46; no voters:
I20260924 10:39:47.482704 4417 raft_consensus.cc:2781] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:39:47.482759 4417 raft_consensus.cc:482] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:39:47.482806 4417 raft_consensus.cc:3037] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:47.483850 4417 raft_consensus.cc:504] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.484197 4417 leader_election.cc:288] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [CANDIDATE]: Term 1 election: Requested vote from peers 8a574b056885428d87c58bc279a93d46 (127.2.8.1:45995), 1b0aadae619d4689a5251f9ab93dce21 (127.2.8.3:40493)
I20260924 10:39:47.484516 3969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ffe2cfbcbf4b9c811f0faa246b49b3" candidate_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a574b056885428d87c58bc279a93d46"
I20260924 10:39:47.484658 3969 raft_consensus.cc:3037] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 8a574b056885428d87c58bc279a93d46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:47.484627 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ffe2cfbcbf4b9c811f0faa246b49b3" candidate_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b0aadae619d4689a5251f9ab93dce21"
I20260924 10:39:47.484745 4249 raft_consensus.cc:3037] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:47.485632 3969 raft_consensus.cc:2454] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 8a574b056885428d87c58bc279a93d46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e56cd06f5a464cb2b2a1f3332ced97 in term 1.
I20260924 10:39:47.485912 4042 leader_election.cc:302] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e56cd06f5a464cb2b2a1f3332ced97, 8a574b056885428d87c58bc279a93d46; no voters:
I20260924 10:39:47.485947 4249 raft_consensus.cc:2454] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e56cd06f5a464cb2b2a1f3332ced97 in term 1.
I20260924 10:39:47.486105 4417 raft_consensus.cc:2781] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:47.486280 4417 raft_consensus.cc:686] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 LEADER]: Becoming Leader. State: Replica: 49e56cd06f5a464cb2b2a1f3332ced97, State: Running, Role: LEADER
I20260924 10:39:47.486440 4417 consensus_queue.cc:237] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } }
I20260924 10:39:47.488251 3812 catalog_manager.cc:5495] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e56cd06f5a464cb2b2a1f3332ced97 (127.2.8.2). New cstate: current_term: 1 leader_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e56cd06f5a464cb2b2a1f3332ced97" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 34367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 } health_report { overall_health: UNKNOWN } } }
I20260924 10:39:47.520414 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:47.516896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.2.8.2:34367)
0924 10:39:47.517052 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:47.517056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:47.517072 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:39:47.517214 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:47.517217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:47.517273 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:47.517364 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:47.517613 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.518420 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.519232 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:47.519558 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:47.519670 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:47.519899 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:47.520000 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:47.520081 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:47.520223 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 10:39:47.520278 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:39:47.528474 3969 raft_consensus.cc:1261] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 8a574b056885428d87c58bc279a93d46 [term 1 FOLLOWER]: Refusing update from remote peer 49e56cd06f5a464cb2b2a1f3332ced97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:47.528535 4249 raft_consensus.cc:1261] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21 [term 1 FOLLOWER]: Refusing update from remote peer 49e56cd06f5a464cb2b2a1f3332ced97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:47.528882 4439 consensus_queue.cc:1035] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b0aadae619d4689a5251f9ab93dce21" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:47.529052 4417 consensus_queue.cc:1035] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a574b056885428d87c58bc279a93d46" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:39:47.543201 4385 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 10:39:49.094967 3810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:39:49.095115 3810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:39:49.097505 3810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d10887d398a64a2d82142a92ed3ca7f9: Sending DeleteTablet for 3 replicas of tablet 95ffe2cfbcbf4b9c811f0faa246b49b3
I20260924 10:39:49.098028 3949 tablet_service.cc:1514] Processing DeleteTablet for tablet 95ffe2cfbcbf4b9c811f0faa246b49b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:49 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:39914
I20260924 10:39:49.098057 4089 tablet_service.cc:1514] Processing DeleteTablet for tablet 95ffe2cfbcbf4b9c811f0faa246b49b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:49 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:51004
I20260924 10:39:49.098078 4228 tablet_service.cc:1514] Processing DeleteTablet for tablet 95ffe2cfbcbf4b9c811f0faa246b49b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:49 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:48772
I20260924 10:39:49.098404 4467 tablet_replica.cc:330] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97: stopping tablet replica
I20260924 10:39:49.098485 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 3884
I20260924 10:39:49.099274 4468 tablet_replica.cc:330] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21: stopping tablet replica
I20260924 10:39:49.099364 4467 raft_consensus.cc:2229] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:39:49.099434 4468 raft_consensus.cc:2229] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:39:49.099543 4468 raft_consensus.cc:2258] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:49.099578 4467 raft_consensus.cc:2258] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:49.100229 4467 ts_tablet_manager.cc:1905] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:49.100265 4468 ts_tablet_manager.cc:1905] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:49.102470 4467 ts_tablet_manager.cc:1918] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:39:49.102567 4467 log.cc:1196] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/95ffe2cfbcbf4b9c811f0faa246b49b3
I20260924 10:39:49.102900 4467 ts_tablet_manager.cc:1939] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 49e56cd06f5a464cb2b2a1f3332ced97: Deleting consensus metadata
I20260924 10:39:49.103509 3797 catalog_manager.cc:4841] TS 49e56cd06f5a464cb2b2a1f3332ced97 (127.2.8.2:34367): tablet 95ffe2cfbcbf4b9c811f0faa246b49b3 (table test-table [id=9d5bd0583e05412b8a8e5a609e31c4fa]) successfully deleted
I20260924 10:39:49.103554 4468 ts_tablet_manager.cc:1918] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 10:39:49.103624 4468 log.cc:1196] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/95ffe2cfbcbf4b9c811f0faa246b49b3
I20260924 10:39:49.103899 4468 ts_tablet_manager.cc:1939] T 95ffe2cfbcbf4b9c811f0faa246b49b3 P 1b0aadae619d4689a5251f9ab93dce21: Deleting consensus metadata
I20260924 10:39:49.105190 3798 catalog_manager.cc:4841] TS 1b0aadae619d4689a5251f9ab93dce21 (127.2.8.3:40493): tablet 95ffe2cfbcbf4b9c811f0faa246b49b3 (table test-table [id=9d5bd0583e05412b8a8e5a609e31c4fa]) successfully deleted
W20260924 10:39:49.106963 3798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.8.1:45995 (error 108)
I20260924 10:39:49.107308 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 4024
I20260924 10:39:49.109119 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:49.108668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:45995 (local address 127.0.0.1:39932)
0924 10:39:49.108909 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:49.109015 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:45995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":95,"threads_started":1}
W20260924 10:39:49.109336 3798 catalog_manager.cc:4584] TS 8a574b056885428d87c58bc279a93d46 (127.2.8.1:45995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 95ffe2cfbcbf4b9c811f0faa246b49b3: Network error: Client connection negotiation failed: client connection to 127.2.8.1:45995: connect: Connection refused (error 111)
I20260924 10:39:49.114183 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 4164
I20260924 10:39:49.120932 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 3780
2026-09-24T10:39:49Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7007 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:49 dist-test-slave-j38h krb5kdc[4473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:49 dist-test-slave-j38h krb5kdc[4473](info): set up 2 sockets
Sep 24 10:39:49 dist-test-slave-j38h krb5kdc[4473](info): commencing operation
krb5kdc: starting...
W20260924 10:39:50.192952 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:39:50 dist-test-slave-j38h krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:39:50.313786 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:36149
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:46359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:36149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:50.423336 4486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:50.423645 4486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:50.423718 4486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:50.427668 4486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:50.427763 4486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:50.427801 4486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:50.427838 4486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:50.427876 4486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:50.433139 4486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:46359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:36149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:50.434209 4486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:50.435278 4486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:39:50.445276 4486 server_base.cc:1034] running on GCE node
W20260924 10:39:50.445282 4492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:50.445483 4491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:50.445276 4494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:50.446055 4486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:50.447525 4486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:50.453040 4486 hybrid_clock.cc:648] HybridClock initialized: now 1790246390453009 us; error 34 us; skew 500 ppm
Sep 24 10:39:50 dist-test-slave-j38h krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246390, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:50.457641 4486 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:39:50.458758 4486 webserver.cc:460] Webserver started at http://127.2.8.62:38087/ using document root <none> and password file <none>
I20260924 10:39:50.459509 4486 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:50.459613 4486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:50.459833 4486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:50.462824 4486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "4dad1867996140169a45ac5ee518565c"
format_stamp: "Formatted at 2026-09-24 10:39:50 on dist-test-slave-j38h"
server_key: "2cca61ac3f20a21df3fab9847a440898"
server_key_iv: "d54d8825f78b406822619444f77caf35"
server_key_version: "encryptionkey@0"
I20260924 10:39:50.463582 4486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "4dad1867996140169a45ac5ee518565c"
format_stamp: "Formatted at 2026-09-24 10:39:50 on dist-test-slave-j38h"
server_key: "2cca61ac3f20a21df3fab9847a440898"
server_key_iv: "d54d8825f78b406822619444f77caf35"
server_key_version: "encryptionkey@0"
I20260924 10:39:50.469107 4486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 10:39:50.472126 4501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:50.473709 4486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:39:50.473862 4486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "4dad1867996140169a45ac5ee518565c"
format_stamp: "Formatted at 2026-09-24 10:39:50 on dist-test-slave-j38h"
server_key: "2cca61ac3f20a21df3fab9847a440898"
server_key_iv: "d54d8825f78b406822619444f77caf35"
server_key_version: "encryptionkey@0"
I20260924 10:39:50.473974 4486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:50.495762 4486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:50.500792 4486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:50.501051 4486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:50.508783 4486 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:36149
I20260924 10:39:50.509132 4553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:36149 every 8 connection(s)
I20260924 10:39:50.509898 4486 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:50.511687 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 4486
I20260924 10:39:50.511840 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:39:50.512122 2080 external_mini_cluster.cc:1428] Setting key 06e04b86150a8837d9d093ae506e22b2
I20260924 10:39:50.512954 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:50.520071 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c: Bootstrap starting.
I20260924 10:39:50.522847 4554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c: Neither blocks nor log segments found. Creating new log.
Sep 24 10:39:50 dist-test-slave-j38h krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:50.523675 4554 log.cc:824] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:50.526187 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c: No bootstrap required, opened a new log
I20260924 10:39:50.530943 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:50.513648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44022 (local address 127.2.8.62:36149)
0924 10:39:50.514084 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:50.514111 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:50.514147 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:39:50.514869 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:50.514892 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:50.515347 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:50.515750 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:50.516362 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:50.518022 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:50.518870 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:50.519625 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:50.519805 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:50.523608 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:50.523615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:50.523633 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:50.523667 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:50.527396 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:50.527878 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:50.527887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:50.527893 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:50.528040 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:50.528361 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:50.528364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:50.528367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:50.529047 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:50.529297 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:50.529551 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 10:39:50.529878 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":167,"threads_started":1}
I20260924 10:39:50.531316 4554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dad1867996140169a45ac5ee518565c" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 36149 } }
I20260924 10:39:50.531597 4554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:50.531659 4554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dad1867996140169a45ac5ee518565c, State: Initialized, Role: FOLLOWER
I20260924 10:39:50.532161 4554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dad1867996140169a45ac5ee518565c" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 36149 } }
I20260924 10:39:50.532330 4554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:50.532405 4554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:50.532513 4554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:50.533795 4554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dad1867996140169a45ac5ee518565c" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 36149 } }
I20260924 10:39:50.534189 4554 leader_election.cc:302] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dad1867996140169a45ac5ee518565c; no voters:
I20260924 10:39:50.534547 4554 leader_election.cc:288] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:50.534653 4559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:50.534885 4559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [term 1 LEADER]: Becoming Leader. State: Replica: 4dad1867996140169a45ac5ee518565c, State: Running, Role: LEADER
I20260924 10:39:50.535197 4559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dad1867996140169a45ac5ee518565c" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 36149 } }
I20260924 10:39:50.535895 4554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:50.536798 4560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dad1867996140169a45ac5ee518565c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dad1867996140169a45ac5ee518565c" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 36149 } } }
I20260924 10:39:50.536931 4560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:50.537338 4561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dad1867996140169a45ac5ee518565c. Latest consensus state: current_term: 1 leader_uuid: "4dad1867996140169a45ac5ee518565c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dad1867996140169a45ac5ee518565c" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 36149 } } }
I20260924 10:39:50.537469 4561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:50.539443 4568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:50.542668 4568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:50.549566 4568 catalog_manager.cc:1347] Generated new cluster ID: bf498b8f69c24fcd95fb510a57c3d151
I20260924 10:39:50.549654 4568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:50.563751 4568 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:50.564719 4568 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:50.588069 4568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dad1867996140169a45ac5ee518565c: Generated new TSK 0
I20260924 10:39:50.588886 4568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:39:50.620779 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:36149
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:46359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:50.731789 4580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:50.732121 4580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:50.732203 4580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:50.736353 4580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:50.736447 4580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:50.736541 4580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:39:50.742182 4580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:46359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:50.743273 4580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:50.744303 4580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:50.753916 4586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:50.753911 4588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:50.754179 4585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:50.754462 4580 server_base.cc:1034] running on GCE node
I20260924 10:39:50.754792 4580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:50.755350 4580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:50.756528 4580 hybrid_clock.cc:648] HybridClock initialized: now 1790246390756501 us; error 36 us; skew 500 ppm
Sep 24 10:39:50 dist-test-slave-j38h krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246390, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:50.760577 4580 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:39:50.761564 4580 webserver.cc:460] Webserver started at http://127.2.8.1:36871/ using document root <none> and password file <none>
I20260924 10:39:50.762290 4580 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:50.762357 4580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:50.762600 4580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:50.764878 4580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "0013b7e8f2b745f789237407fced1445"
format_stamp: "Formatted at 2026-09-24 10:39:50 on dist-test-slave-j38h"
server_key: "f8b6d670fd241f45030b30a6d4130ae6"
server_key_iv: "f7c4b199c073a5e3e0856749f92c4877"
server_key_version: "encryptionkey@0"
I20260924 10:39:50.765530 4580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "0013b7e8f2b745f789237407fced1445"
format_stamp: "Formatted at 2026-09-24 10:39:50 on dist-test-slave-j38h"
server_key: "f8b6d670fd241f45030b30a6d4130ae6"
server_key_iv: "f7c4b199c073a5e3e0856749f92c4877"
server_key_version: "encryptionkey@0"
I20260924 10:39:50.769474 4580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:39:50.771946 4595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:50.773289 4580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:39:50.773435 4580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "0013b7e8f2b745f789237407fced1445"
format_stamp: "Formatted at 2026-09-24 10:39:50 on dist-test-slave-j38h"
server_key: "f8b6d670fd241f45030b30a6d4130ae6"
server_key_iv: "f7c4b199c073a5e3e0856749f92c4877"
server_key_version: "encryptionkey@0"
I20260924 10:39:50.773551 4580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:50.802788 4580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:50.808379 4580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:50.808597 4580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:50.809279 4580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:50.810133 4580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:50.810205 4580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:50.810276 4580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:50.810338 4580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:50.819278 4580 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:33881
I20260924 10:39:50.819301 4708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:33881 every 8 connection(s)
I20260924 10:39:50.820194 4580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:39:50.827859 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 4580
I20260924 10:39:50.828004 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:39:50.828296 2080 external_mini_cluster.cc:1428] Setting key d29cfc5ad70e356f29211a8cfe3920cc
Sep 24 10:39:50 dist-test-slave-j38h krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246390, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:50.837551 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:50.822055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:36531 (local address 127.2.8.62:36149)
0924 10:39:50.822239 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:50.822242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:50.823095 (+ 853us) server_negotiation.cc:407] Connection header received
0924 10:39:50.824522 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:50.824525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:50.824581 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:50.824661 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:50.825865 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:50.826399 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:50.827528 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:50.827764 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:50.827885 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:50.831745 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:50.831749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:50.831751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:50.831775 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:50.834931 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:50.835665 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:50.835673 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:50.835677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:50.835778 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:50.836146 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:50.836152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:50.836156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:50.836439 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:50.836583 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:50.837182 (+ 599us) server_negotiation.cc:299] Negotiation successful
0924 10:39:50.837340 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 10:39:50.838449 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:50.822468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:36149 (local address 127.2.8.1:36531)
0924 10:39:50.822933 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:50.822976 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:50.824305 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:50.824828 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:50.824834 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:50.825233 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:50.825701 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:50.825713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:50.826523 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:50.826527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:50.827404 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:50.827412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:50.827869 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:50.827921 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:50.828145 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:50.829557 (+ 1412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:50.829579 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:50.831554 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:50.835103 (+ 3549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:50.835112 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:50.835133 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:50.835558 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:50.835883 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:50.835885 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:50.835887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:50.836023 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:50.836588 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:50.836593 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:50.836877 (+ 284us) client_negotiation.cc:769] Sending connection context
0924 10:39:50.837114 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 10:39:50.837518 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
I20260924 10:39:50.839753 4709 heartbeater.cc:347] Connected to a master server at 127.2.8.62:36149
I20260924 10:39:50.840024 4709 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:50.840574 4709 heartbeater.cc:511] Master 127.2.8.62:36149 requested a full tablet report, sending...
I20260924 10:39:50.842088 4518 ts_manager.cc:194] Registered new tserver with Master: 0013b7e8f2b745f789237407fced1445 (127.2.8.1:33881)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
I20260924 10:39:50.843726 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:36531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:39:50.859987 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:36149
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:46359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:50.967093 4714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:50.967371 4714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:50.967475 4714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:50.971385 4714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:50.971488 4714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:50.971580 4714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:39:50.976642 4714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:46359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:50.977712 4714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:50.978801 4714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:50.988279 4722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:50.988284 4720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:50.988453 4714 server_base.cc:1034] running on GCE node
W20260924 10:39:50.988283 4719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:50.988853 4714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:50.989513 4714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:50.990695 4714 hybrid_clock.cc:648] HybridClock initialized: now 1790246390990666 us; error 44 us; skew 500 ppm
Sep 24 10:39:50 dist-test-slave-j38h krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246390, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:50.994194 4714 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:39:50.995014 4714 webserver.cc:460] Webserver started at http://127.2.8.2:40023/ using document root <none> and password file <none>
I20260924 10:39:50.995721 4714 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:50.995810 4714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:50.996038 4714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:50.998104 4714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "71fac1929bf5448d872b9896ae8ea43e"
format_stamp: "Formatted at 2026-09-24 10:39:50 on dist-test-slave-j38h"
server_key: "5125df3dafcb5bc1b25bf4ab83148b5a"
server_key_iv: "675673e0cb324f23e8dcdaec36d90f05"
server_key_version: "encryptionkey@0"
I20260924 10:39:50.998661 4714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "71fac1929bf5448d872b9896ae8ea43e"
format_stamp: "Formatted at 2026-09-24 10:39:50 on dist-test-slave-j38h"
server_key: "5125df3dafcb5bc1b25bf4ab83148b5a"
server_key_iv: "675673e0cb324f23e8dcdaec36d90f05"
server_key_version: "encryptionkey@0"
I20260924 10:39:51.002250 4714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:39:51.004586 4729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:51.006069 4714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:39:51.006215 4714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "71fac1929bf5448d872b9896ae8ea43e"
format_stamp: "Formatted at 2026-09-24 10:39:50 on dist-test-slave-j38h"
server_key: "5125df3dafcb5bc1b25bf4ab83148b5a"
server_key_iv: "675673e0cb324f23e8dcdaec36d90f05"
server_key_version: "encryptionkey@0"
I20260924 10:39:51.006335 4714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:51.027140 4714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:51.032393 4714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:51.032548 4714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:51.033098 4714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:51.033856 4714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:51.033907 4714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:51.033948 4714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:51.033970 4714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:51.042627 4714 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:34325
I20260924 10:39:51.042652 4842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:34325 every 8 connection(s)
I20260924 10:39:51.043509 4714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:39:51.046254 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 4714
I20260924 10:39:51.046355 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:39:51.046612 2080 external_mini_cluster.cc:1428] Setting key 7b0ff51785e171eb9871de81a93ea170
Sep 24 10:39:51 dist-test-slave-j38h krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246390, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
I20260924 10:39:51.059862 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:51.045290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:51305 (local address 127.2.8.62:36149)
0924 10:39:51.045398 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:51.045402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:51.046237 (+ 835us) server_negotiation.cc:407] Connection header received
0924 10:39:51.047782 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:51.047785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:51.047831 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:51.047928 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:51.049507 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:51.050357 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:51.051469 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:51.051688 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:51.051797 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:51.054844 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:51.054847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:51.054850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:51.054877 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:51.057825 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:51.058334 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:51.058339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:51.058341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:51.058430 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:51.058781 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:51.058786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:51.058788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:51.058972 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:51.059076 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:51.059569 (+ 493us) server_negotiation.cc:299] Negotiation successful
0924 10:39:51.059685 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 10:39:51.060655 4845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:51.045677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:36149 (local address 127.2.8.2:51305)
0924 10:39:51.046101 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:51.046140 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:51.047557 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:51.048132 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:51.048142 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:51.048635 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:51.049347 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:51.049369 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:51.050484 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:51.050486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:51.051333 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:51.051341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:51.051783 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:51.051834 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:51.052007 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:51.053328 (+ 1321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:51.053343 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:51.054688 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:51.057965 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:51.057971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:51.057986 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:51.058231 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:51.058529 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:51.058533 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:51.058536 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:51.058688 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:51.059073 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:51.059080 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:51.059330 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 10:39:51.059514 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 10:39:51.059824 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
I20260924 10:39:51.061829 4843 heartbeater.cc:347] Connected to a master server at 127.2.8.62:36149
I20260924 10:39:51.062088 4843 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:51.062580 4843 heartbeater.cc:511] Master 127.2.8.62:36149 requested a full tablet report, sending...
I20260924 10:39:51.063570 4518 ts_manager.cc:194] Registered new tserver with Master: 71fac1929bf5448d872b9896ae8ea43e (127.2.8.2:34325)
I20260924 10:39:51.064162 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:51305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:39:51.074275 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:36149
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:46359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:51.182365 4848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:51.182603 4848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:51.182642 4848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:51.186589 4848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:51.186635 4848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:51.186719 4848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:39:51.191877 4848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:46359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:51.192868 4848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:51.193832 4848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:51.202939 4856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:51.202939 4854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:51.202989 4853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:51.203274 4848 server_base.cc:1034] running on GCE node
I20260924 10:39:51.203635 4848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:51.204246 4848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:51.205420 4848 hybrid_clock.cc:648] HybridClock initialized: now 1790246391205385 us; error 42 us; skew 500 ppm
Sep 24 10:39:51 dist-test-slave-j38h krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246391, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:51.208680 4848 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:39:51.209614 4848 webserver.cc:460] Webserver started at http://127.2.8.3:36937/ using document root <none> and password file <none>
I20260924 10:39:51.210332 4848 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:51.210427 4848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:51.210626 4848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:51.212589 4848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "32ee7d93c5bb42d49a4426bd30564fe1"
format_stamp: "Formatted at 2026-09-24 10:39:51 on dist-test-slave-j38h"
server_key: "3e9eea4593aa2df06f577590ea6dee0d"
server_key_iv: "56c8393c79bd420cc4baf739e508b882"
server_key_version: "encryptionkey@0"
I20260924 10:39:51.213191 4848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "32ee7d93c5bb42d49a4426bd30564fe1"
format_stamp: "Formatted at 2026-09-24 10:39:51 on dist-test-slave-j38h"
server_key: "3e9eea4593aa2df06f577590ea6dee0d"
server_key_iv: "56c8393c79bd420cc4baf739e508b882"
server_key_version: "encryptionkey@0"
I20260924 10:39:51.216683 4848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:39:51.219122 4863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:51.220402 4848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:51.220541 4848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "32ee7d93c5bb42d49a4426bd30564fe1"
format_stamp: "Formatted at 2026-09-24 10:39:51 on dist-test-slave-j38h"
server_key: "3e9eea4593aa2df06f577590ea6dee0d"
server_key_iv: "56c8393c79bd420cc4baf739e508b882"
server_key_version: "encryptionkey@0"
I20260924 10:39:51.220645 4848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:51.249933 4848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:51.254280 4848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:51.254487 4848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:51.255065 4848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:51.255867 4848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:51.255934 4848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:51.256003 4848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:51.256052 4848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:51.265105 4848 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:35515
I20260924 10:39:51.265117 4976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:35515 every 8 connection(s)
I20260924 10:39:51.265954 4848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:39:51.270388 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 4848
I20260924 10:39:51.270515 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:39:51.270768 2080 external_mini_cluster.cc:1428] Setting key 14b4c06fb98007da457d5fbac047c427
Sep 24 10:39:51 dist-test-slave-j38h krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246391, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:51.281433 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:51.267786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:41575 (local address 127.2.8.62:36149)
0924 10:39:51.267931 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:51.267939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:51.268681 (+ 742us) server_negotiation.cc:407] Connection header received
0924 10:39:51.270141 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:51.270145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:51.270207 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:51.270313 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:51.271686 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:51.272389 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:51.273737 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:51.273966 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:51.274076 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:51.276914 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:51.276917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:51.276919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:51.276940 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:51.279337 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:51.279816 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:51.279819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:51.279836 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:51.279905 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:51.280245 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:51.280247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:51.280248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:51.280418 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:51.280496 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:51.281220 (+ 724us) server_negotiation.cc:299] Negotiation successful
0924 10:39:51.281313 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:39:51.282212 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:51.268115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:36149 (local address 127.2.8.3:41575)
0924 10:39:51.268535 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:51.268574 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:51.269926 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:51.270502 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:51.270513 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:51.270964 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:51.271520 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:51.271557 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:51.272539 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:51.272542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:51.273614 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:51.273623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:51.274063 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:51.274112 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:51.274282 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:51.275311 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:51.275327 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:51.276741 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:51.279466 (+ 2725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:51.279470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:51.279481 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:51.279723 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:51.280021 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:51.280024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:51.280025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:51.280156 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:51.280513 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:51.280517 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:51.280951 (+ 434us) client_negotiation.cc:769] Sending connection context
0924 10:39:51.281172 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 10:39:51.281419 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":102,"threads_started":1}
I20260924 10:39:51.283376 4977 heartbeater.cc:347] Connected to a master server at 127.2.8.62:36149
I20260924 10:39:51.283612 4977 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:51.284184 4977 heartbeater.cc:511] Master 127.2.8.62:36149 requested a full tablet report, sending...
I20260924 10:39:51.285249 4518 ts_manager.cc:194] Registered new tserver with Master: 32ee7d93c5bb42d49a4426bd30564fe1 (127.2.8.3:35515)
I20260924 10:39:51.285789 4518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:41575
I20260924 10:39:51.296319 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 10:39:51.308482 4557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:39:51.307100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.2.8.62:36149)
0924 10:39:51.307248 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:51.307253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:51.307443 (+ 190us) server_negotiation.cc:407] Connection header received
0924 10:39:51.307534 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:51.307537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:51.307579 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:51.307649 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:51.308340 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44030: BlockingRecv error: recv got EOF from 127.0.0.1:44030 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 10:39:51.309799 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 4580
I20260924 10:39:51.315361 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 4714
I20260924 10:39:51.320375 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 4848
I20260924 10:39:51.325311 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 4486
2026-09-24T10:39:51Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2202 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:51 dist-test-slave-j38h krb5kdc[4989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:51 dist-test-slave-j38h krb5kdc[4989](info): set up 2 sockets
Sep 24 10:39:51 dist-test-slave-j38h krb5kdc[4989](info): commencing operation
krb5kdc: starting...
W20260924 10:39:52.369182 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:39:52 dist-test-slave-j38h krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:39:52.490533 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:41815
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:41815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:52.600589 5002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:52.600863 5002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:52.600903 5002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:52.604831 5002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:52.604894 5002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:52.604913 5002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:52.604930 5002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:52.604945 5002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:52.610157 5002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:41815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:41815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:52.611155 5002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:52.612119 5002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:52.620863 5010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:52.620846 5002 server_base.cc:1034] running on GCE node
W20260924 10:39:52.620853 5007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:52.620877 5008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:52.621693 5002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:52.622792 5002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:52.623970 5002 hybrid_clock.cc:648] HybridClock initialized: now 1790246392623941 us; error 43 us; skew 500 ppm
Sep 24 10:39:52 dist-test-slave-j38h krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246392, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:52.633369 5002 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:39:52.634586 5002 webserver.cc:460] Webserver started at http://127.2.8.62:37515/ using document root <none> and password file <none>
I20260924 10:39:52.635198 5002 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:52.635296 5002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:52.635560 5002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:52.637696 5002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "e7ff0afb4f5a4a52b81c3444639f7859"
format_stamp: "Formatted at 2026-09-24 10:39:52 on dist-test-slave-j38h"
server_key: "2b4e2f7bdb6a23f1cec1b1432a85308f"
server_key_iv: "83253a0a855315aff2269493f1318a22"
server_key_version: "encryptionkey@0"
I20260924 10:39:52.638259 5002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "e7ff0afb4f5a4a52b81c3444639f7859"
format_stamp: "Formatted at 2026-09-24 10:39:52 on dist-test-slave-j38h"
server_key: "2b4e2f7bdb6a23f1cec1b1432a85308f"
server_key_iv: "83253a0a855315aff2269493f1318a22"
server_key_version: "encryptionkey@0"
I20260924 10:39:52.641966 5002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:39:52.644331 5017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:52.645601 5002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:52.645756 5002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "e7ff0afb4f5a4a52b81c3444639f7859"
format_stamp: "Formatted at 2026-09-24 10:39:52 on dist-test-slave-j38h"
server_key: "2b4e2f7bdb6a23f1cec1b1432a85308f"
server_key_iv: "83253a0a855315aff2269493f1318a22"
server_key_version: "encryptionkey@0"
I20260924 10:39:52.645862 5002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:52.661355 5002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:52.664057 5002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:52.664258 5002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:52.671185 5002 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:41815
I20260924 10:39:52.671509 5069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:41815 every 8 connection(s)
I20260924 10:39:52.672134 5002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:52.674834 5070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:52.677227 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 5002
I20260924 10:39:52.677342 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:39:52.677635 2080 external_mini_cluster.cc:1428] Setting key 01640551f14009dbe4eb9b6900af1aa5
I20260924 10:39:52.680951 5070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859: Bootstrap starting.
I20260924 10:39:52.683399 5070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:52.684182 5070 log.cc:824] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:39:52 dist-test-slave-j38h krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:52.686371 5070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859: No bootstrap required, opened a new log
I20260924 10:39:52.690629 5070 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ff0afb4f5a4a52b81c3444639f7859" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41815 } }
I20260924 10:39:52.690842 5070 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:52.690905 5070 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff0afb4f5a4a52b81c3444639f7859, State: Initialized, Role: FOLLOWER
I20260924 10:39:52.691360 5070 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ff0afb4f5a4a52b81c3444639f7859" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41815 } }
I20260924 10:39:52.691484 5070 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:52.691567 5070 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:52.691569 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:52.678899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.2.8.62:41815)
0924 10:39:52.679289 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:52.679304 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:52.679337 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:39:52.679993 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:52.680009 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:52.680324 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:52.680592 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:52.681515 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:52.682478 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:52.683621 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:52.684086 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:52.684198 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:52.686997 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:52.687002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:52.687013 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:52.687038 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:52.688675 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:52.689171 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:52.689176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:52.689180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:52.689284 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:52.689566 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:52.689568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:52.689569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:52.689937 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:52.690083 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:52.690355 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 10:39:52.690698 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":119,"threads_started":1}
I20260924 10:39:52.691694 5070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:52.692728 5070 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ff0afb4f5a4a52b81c3444639f7859" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41815 } }
I20260924 10:39:52.693107 5070 leader_election.cc:302] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7ff0afb4f5a4a52b81c3444639f7859; no voters:
I20260924 10:39:52.693442 5070 leader_election.cc:288] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:52.693511 5075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:52.693706 5075 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [term 1 LEADER]: Becoming Leader. State: Replica: e7ff0afb4f5a4a52b81c3444639f7859, State: Running, Role: LEADER
I20260924 10:39:52.694006 5075 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ff0afb4f5a4a52b81c3444639f7859" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41815 } }
I20260924 10:39:52.694351 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:39:52.694682 5070 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:52.695101 5075 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7ff0afb4f5a4a52b81c3444639f7859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ff0afb4f5a4a52b81c3444639f7859" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41815 } } }
I20260924 10:39:52.695209 5075 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:52.695652 5076 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7ff0afb4f5a4a52b81c3444639f7859. Latest consensus state: current_term: 1 leader_uuid: "e7ff0afb4f5a4a52b81c3444639f7859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ff0afb4f5a4a52b81c3444639f7859" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41815 } } }
I20260924 10:39:52.695768 5076 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859 [sys.catalog]: This master's current role is: LEADER
W20260924 10:39:52.699504 5091 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:39:52.699580 5091 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:39:52.699638 5079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:52.700482 5079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:52.706609 5079 catalog_manager.cc:1347] Generated new cluster ID: a50ac0fd94cd4a908b485bb4e081c3df
I20260924 10:39:52.706671 5079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:52.712687 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:52.707118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.2.8.62:41815)
0924 10:39:52.707228 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:52.707232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:52.707592 (+ 360us) server_negotiation.cc:407] Connection header received
0924 10:39:52.707717 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:52.707720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:52.707762 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:52.707838 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:52.708440 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:52.709211 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:52.710386 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:52.710744 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:52.710825 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:52.711750 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:52.711754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:52.711757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:52.712251 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:52.712283 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:52.712301 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:52.712392 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:52.712498 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 10:39:52.712553 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 10:39:52.726133 5079 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:52.727362 5079 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:52.739287 5079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7ff0afb4f5a4a52b81c3444639f7859: Generated new TSK 0
I20260924 10:39:52.740139 5079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:52.753904 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 5002
2026-09-24T10:39:52Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1426 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:52 dist-test-slave-j38h krb5kdc[5103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:52 dist-test-slave-j38h krb5kdc[5103](info): set up 2 sockets
Sep 24 10:39:52 dist-test-slave-j38h krb5kdc[5103](info): commencing operation
krb5kdc: starting...
W20260924 10:39:53.785687 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:39:53 dist-test-slave-j38h krb5kdc[5103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:39:53.912139 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:43223
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:43223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:54.024472 5116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:54.024813 5116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:54.024901 5116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:54.028908 5116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:54.029042 5116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:54.029093 5116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:54.029134 5116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:54.029174 5116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:54.034353 5116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:43223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:43223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:54.035437 5116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:54.036487 5116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:54.044915 5121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:54.044872 5122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:54.044883 5124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:54.045214 5116 server_base.cc:1034] running on GCE node
I20260924 10:39:54.045833 5116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:54.046973 5116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:54.048174 5116 hybrid_clock.cc:648] HybridClock initialized: now 1790246394048155 us; error 41 us; skew 500 ppm
Sep 24 10:39:54 dist-test-slave-j38h krb5kdc[5103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246394, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:54.051076 5116 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:39:54.051968 5116 webserver.cc:460] Webserver started at http://127.2.8.62:37385/ using document root <none> and password file <none>
I20260924 10:39:54.052541 5116 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:54.052605 5116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:54.052839 5116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:54.054966 5116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "adb633ad6e004203bc11b1b2a520b682"
format_stamp: "Formatted at 2026-09-24 10:39:54 on dist-test-slave-j38h"
server_key: "7f25011b6485dd1556c47f83cddda4f7"
server_key_iv: "f4b0fa5c34d1d0cd45de958b3b1f5b64"
server_key_version: "encryptionkey@0"
I20260924 10:39:54.055568 5116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "adb633ad6e004203bc11b1b2a520b682"
format_stamp: "Formatted at 2026-09-24 10:39:54 on dist-test-slave-j38h"
server_key: "7f25011b6485dd1556c47f83cddda4f7"
server_key_iv: "f4b0fa5c34d1d0cd45de958b3b1f5b64"
server_key_version: "encryptionkey@0"
I20260924 10:39:54.059506 5116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 10:39:54.062073 5131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:54.063421 5116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260924 10:39:54.063558 5116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "adb633ad6e004203bc11b1b2a520b682"
format_stamp: "Formatted at 2026-09-24 10:39:54 on dist-test-slave-j38h"
server_key: "7f25011b6485dd1556c47f83cddda4f7"
server_key_iv: "f4b0fa5c34d1d0cd45de958b3b1f5b64"
server_key_version: "encryptionkey@0"
I20260924 10:39:54.063673 5116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:54.082018 5116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:54.085283 5116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:54.085498 5116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:54.092564 5116 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:43223
I20260924 10:39:54.092869 5183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:43223 every 8 connection(s)
I20260924 10:39:54.093559 5116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:54.096482 5184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:54.098423 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 5116
I20260924 10:39:54.098515 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:39:54.098775 2080 external_mini_cluster.cc:1428] Setting key 550f2b314eaff73f7cee55a9e7f78edd
I20260924 10:39:54.103626 5184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682: Bootstrap starting.
I20260924 10:39:54.106551 5184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:54.107353 5184 log.cc:824] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:39:54 dist-test-slave-j38h krb5kdc[5103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:54.109668 5184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682: No bootstrap required, opened a new log
I20260924 10:39:54.115039 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:54.100183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36442 (local address 127.2.8.62:43223)
0924 10:39:54.100597 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:54.100617 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:54.100661 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:39:54.101601 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:54.101627 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:54.102087 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:54.102481 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:54.103151 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:54.104725 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:54.105682 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:54.106430 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:54.106620 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:54.109483 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:54.109489 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:54.109506 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:54.109537 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:54.111649 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:54.112098 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:54.112106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:54.112115 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:54.112304 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:54.112595 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:54.112601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:54.112605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:54.113190 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:54.113394 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:54.113667 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 10:39:54.113988 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":107,"threads_started":1}
I20260924 10:39:54.115024 5184 raft_consensus.cc:348] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb633ad6e004203bc11b1b2a520b682" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43223 } }
I20260924 10:39:54.115339 5184 raft_consensus.cc:374] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:54.115403 5184 raft_consensus.cc:729] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb633ad6e004203bc11b1b2a520b682, State: Initialized, Role: FOLLOWER
I20260924 10:39:54.115913 5184 consensus_queue.cc:260] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb633ad6e004203bc11b1b2a520b682" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43223 } }
I20260924 10:39:54.116076 5184 raft_consensus.cc:388] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:54.116148 5184 raft_consensus.cc:482] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:54.116259 5184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:54.117549 5184 raft_consensus.cc:504] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb633ad6e004203bc11b1b2a520b682" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43223 } }
I20260924 10:39:54.117918 5184 leader_election.cc:302] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adb633ad6e004203bc11b1b2a520b682; no voters:
I20260924 10:39:54.118245 5184 leader_election.cc:288] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:54.118438 5189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:54.118680 5189 raft_consensus.cc:686] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [term 1 LEADER]: Becoming Leader. State: Replica: adb633ad6e004203bc11b1b2a520b682, State: Running, Role: LEADER
I20260924 10:39:54.118999 5189 consensus_queue.cc:237] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb633ad6e004203bc11b1b2a520b682" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43223 } }
I20260924 10:39:54.119542 5184 sys_catalog.cc:564] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:54.120756 5191 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adb633ad6e004203bc11b1b2a520b682. Latest consensus state: current_term: 1 leader_uuid: "adb633ad6e004203bc11b1b2a520b682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb633ad6e004203bc11b1b2a520b682" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43223 } } }
I20260924 10:39:54.120839 5190 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adb633ad6e004203bc11b1b2a520b682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb633ad6e004203bc11b1b2a520b682" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43223 } } }
I20260924 10:39:54.120899 5191 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:54.120932 5190 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:54.121937 5198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:54.124739 5198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:54.130904 5198 catalog_manager.cc:1347] Generated new cluster ID: 38114593f8a643dfabedd1cad4af363b
I20260924 10:39:54.130977 5198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:54.160902 5198 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:54.162216 5198 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:54.171492 5198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adb633ad6e004203bc11b1b2a520b682: Generated new TSK 0
I20260924 10:39:54.172318 5198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:39:54.207911 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43223
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:54.317682 5210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:54.317931 5210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:54.317973 5210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:54.322353 5210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:54.322408 5210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:54.322501 5210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:39:54.327564 5210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:54.328580 5210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:54.329598 5210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:54.338943 5216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:54.338923 5215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:54.339026 5210 server_base.cc:1034] running on GCE node
W20260924 10:39:54.338923 5218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:54.339531 5210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:54.340174 5210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:54.341348 5210 hybrid_clock.cc:648] HybridClock initialized: now 1790246394341326 us; error 33 us; skew 500 ppm
Sep 24 10:39:54 dist-test-slave-j38h krb5kdc[5103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246394, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:54.344188 5210 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:39:54.345039 5210 webserver.cc:460] Webserver started at http://127.2.8.1:33455/ using document root <none> and password file <none>
I20260924 10:39:54.345566 5210 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:54.345649 5210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:54.345867 5210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:54.347790 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "d2f0a61d510941ff853491c11b595bcb"
format_stamp: "Formatted at 2026-09-24 10:39:54 on dist-test-slave-j38h"
server_key: "b9cecbcb37e23c80e082ba9b22fca7dc"
server_key_iv: "dd9b60cd4e6d0958759241f5a31ff55f"
server_key_version: "encryptionkey@0"
I20260924 10:39:54.348337 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "d2f0a61d510941ff853491c11b595bcb"
format_stamp: "Formatted at 2026-09-24 10:39:54 on dist-test-slave-j38h"
server_key: "b9cecbcb37e23c80e082ba9b22fca7dc"
server_key_iv: "dd9b60cd4e6d0958759241f5a31ff55f"
server_key_version: "encryptionkey@0"
I20260924 10:39:54.351967 5210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:39:54.354430 5225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:54.355650 5210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:39:54.355780 5210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "d2f0a61d510941ff853491c11b595bcb"
format_stamp: "Formatted at 2026-09-24 10:39:54 on dist-test-slave-j38h"
server_key: "b9cecbcb37e23c80e082ba9b22fca7dc"
server_key_iv: "dd9b60cd4e6d0958759241f5a31ff55f"
server_key_version: "encryptionkey@0"
I20260924 10:39:54.355893 5210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:54.377206 5210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:54.380568 5210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:54.380776 5210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:54.381439 5210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:54.382282 5210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:54.382351 5210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:54.382417 5210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:54.382467 5210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:54.391237 5210 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:46333
I20260924 10:39:54.391261 5338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:46333 every 8 connection(s)
I20260924 10:39:54.392099 5210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:39:54.394443 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 5210
I20260924 10:39:54.394574 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:39:54.394912 2080 external_mini_cluster.cc:1428] Setting key 93e4e1e11dc816aacaa890b108d68df6
Sep 24 10:39:54 dist-test-slave-j38h krb5kdc[5103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246394, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
I20260924 10:39:54.407097 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:54.393849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:60579 (local address 127.2.8.62:43223)
0924 10:39:54.393982 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:54.393986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:54.394893 (+ 907us) server_negotiation.cc:407] Connection header received
0924 10:39:54.396523 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:54.396525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:54.396566 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:54.396645 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:54.398275 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:54.399140 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:54.400258 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:54.400687 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:54.400803 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:54.403565 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:54.403568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:54.403570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:54.403593 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:54.405064 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:54.405542 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:54.405546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:54.405548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:54.405624 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:54.405950 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:54.405955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:54.405957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:54.406129 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:54.406234 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:54.406797 (+ 563us) server_negotiation.cc:299] Negotiation successful
0924 10:39:54.406927 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
I20260924 10:39:54.407928 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:54.394210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43223 (local address 127.2.8.1:60579)
0924 10:39:54.394698 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:54.394758 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:54.396304 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:54.396868 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:54.396881 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:54.397402 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:54.398113 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:54.398137 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:54.399264 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:54.399267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:54.400140 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:54.400148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:54.400791 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:54.400842 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:54.401042 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:54.401681 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:54.401709 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:54.403400 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:54.405169 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:54.405175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:54.405189 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:54.405440 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:54.405721 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:54.405724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:54.405726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:54.405856 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:54.406230 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:54.406236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:54.406514 (+ 278us) client_negotiation.cc:769] Sending connection context
0924 10:39:54.406714 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 10:39:54.407030 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":111,"threads_started":1}
I20260924 10:39:54.409197 5339 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43223
I20260924 10:39:54.409436 5339 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:54.409901 5339 heartbeater.cc:511] Master 127.2.8.62:43223 requested a full tablet report, sending...
I20260924 10:39:54.411319 5148 ts_manager.cc:194] Registered new tserver with Master: d2f0a61d510941ff853491c11b595bcb (127.2.8.1:46333)
I20260924 10:39:54.412405 5148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:60579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:39:54.424711 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43223
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:54.533821 5344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:54.534085 5344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:54.534126 5344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:54.538012 5344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:54.538067 5344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:54.538122 5344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:39:54.543286 5344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:54.544234 5344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:54.545197 5344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:54.554278 5349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:54.554282 5350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:54.554587 5352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:54.554608 5344 server_base.cc:1034] running on GCE node
I20260924 10:39:54.555075 5344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:54.555634 5344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:54.556825 5344 hybrid_clock.cc:648] HybridClock initialized: now 1790246394556810 us; error 34 us; skew 500 ppm
Sep 24 10:39:54 dist-test-slave-j38h krb5kdc[5103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246394, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:54.560560 5344 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:39:54.561471 5344 webserver.cc:460] Webserver started at http://127.2.8.2:37157/ using document root <none> and password file <none>
I20260924 10:39:54.562047 5344 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:54.562135 5344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:54.562345 5344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:54.564335 5344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "e97cdd09efb443d7b048afe34ec53d23"
format_stamp: "Formatted at 2026-09-24 10:39:54 on dist-test-slave-j38h"
server_key: "60ef13d28a76a73d9b3a7f9e824ed347"
server_key_iv: "550d2c2010d5d5240fd69d5e90a65ada"
server_key_version: "encryptionkey@0"
I20260924 10:39:54.564903 5344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "e97cdd09efb443d7b048afe34ec53d23"
format_stamp: "Formatted at 2026-09-24 10:39:54 on dist-test-slave-j38h"
server_key: "60ef13d28a76a73d9b3a7f9e824ed347"
server_key_iv: "550d2c2010d5d5240fd69d5e90a65ada"
server_key_version: "encryptionkey@0"
I20260924 10:39:54.568531 5344 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:39:54.571156 5359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:54.572368 5344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:54.572505 5344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "e97cdd09efb443d7b048afe34ec53d23"
format_stamp: "Formatted at 2026-09-24 10:39:54 on dist-test-slave-j38h"
server_key: "60ef13d28a76a73d9b3a7f9e824ed347"
server_key_iv: "550d2c2010d5d5240fd69d5e90a65ada"
server_key_version: "encryptionkey@0"
I20260924 10:39:54.572611 5344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:54.593679 5344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:54.596830 5344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:54.597074 5344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:54.597661 5344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:54.598474 5344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:54.598541 5344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:54.598611 5344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:54.598665 5344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:54.607558 5344 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:36869
I20260924 10:39:54.607586 5472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:36869 every 8 connection(s)
I20260924 10:39:54.608516 5344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:39:54.610924 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 5344
I20260924 10:39:54.611024 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:39:54.611317 2080 external_mini_cluster.cc:1428] Setting key 4ac539f8a05c8d17b11055b4a864f96d
Sep 24 10:39:54 dist-test-slave-j38h krb5kdc[5103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246394, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
I20260924 10:39:54.624848 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:54.610387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:52503 (local address 127.2.8.62:43223)
0924 10:39:54.610563 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:54.610567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:54.611470 (+ 903us) server_negotiation.cc:407] Connection header received
0924 10:39:54.612812 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:54.612816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:54.612866 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:54.612960 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:54.614243 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:54.615103 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:54.616442 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:54.616857 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:54.617018 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:54.620278 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:54.620281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:54.620283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:54.620311 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:54.622268 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:54.622890 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:54.622896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:54.622899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:54.622996 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:54.623421 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:54.623425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:54.623427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:54.623729 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:54.623856 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:54.624392 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 10:39:54.624555 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 10:39:54.625494 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:54.610809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43223 (local address 127.2.8.2:52503)
0924 10:39:54.611310 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:54.611372 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:54.612623 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:54.613165 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:54.613172 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:54.613559 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:54.614057 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:54.614080 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:54.615283 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:54.615288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:54.616313 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:54.616322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:54.617137 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:54.617189 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:54.617341 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:54.618249 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:54.618277 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:54.620051 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:54.622426 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:54.622434 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:54.622453 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:54.622766 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:54.623110 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:54.623115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:54.623118 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:54.623289 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:54.623857 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:54.623863 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:54.624117 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 10:39:54.624286 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 10:39:54.624565 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":142,"threads_started":1}
I20260924 10:39:54.626789 5473 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43223
I20260924 10:39:54.627179 5473 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:54.627767 5473 heartbeater.cc:511] Master 127.2.8.62:43223 requested a full tablet report, sending...
I20260924 10:39:54.628917 5148 ts_manager.cc:194] Registered new tserver with Master: e97cdd09efb443d7b048afe34ec53d23 (127.2.8.2:36869)
I20260924 10:39:54.629544 5148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:52503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:39:54.640842 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43223
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:54.757085 5478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:54.757310 5478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:54.757391 5478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:54.761489 5478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:54.761540 5478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:54.761627 5478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:39:54.766631 5478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:54.767603 5478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:54.768572 5478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:54.777743 5484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:54.777738 5486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:54.777681 5483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:54.777900 5478 server_base.cc:1034] running on GCE node
I20260924 10:39:54.778398 5478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:54.779058 5478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:54.780242 5478 hybrid_clock.cc:648] HybridClock initialized: now 1790246394780222 us; error 34 us; skew 500 ppm
Sep 24 10:39:54 dist-test-slave-j38h krb5kdc[5103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246394, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:54.783082 5478 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:39:54.783936 5478 webserver.cc:460] Webserver started at http://127.2.8.3:45215/ using document root <none> and password file <none>
I20260924 10:39:54.784467 5478 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:54.784523 5478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:54.784679 5478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:54.786653 5478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "02978715d8dd4732a76623602baec2b4"
format_stamp: "Formatted at 2026-09-24 10:39:54 on dist-test-slave-j38h"
server_key: "c5fe308bc05d08402cb0709fd8cf60fb"
server_key_iv: "94cf6725b5cc2051b20134aeb2b62e6b"
server_key_version: "encryptionkey@0"
I20260924 10:39:54.787138 5478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "02978715d8dd4732a76623602baec2b4"
format_stamp: "Formatted at 2026-09-24 10:39:54 on dist-test-slave-j38h"
server_key: "c5fe308bc05d08402cb0709fd8cf60fb"
server_key_iv: "94cf6725b5cc2051b20134aeb2b62e6b"
server_key_version: "encryptionkey@0"
I20260924 10:39:54.790802 5478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 10:39:54.793233 5493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:54.794476 5478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 10:39:54.794586 5478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "02978715d8dd4732a76623602baec2b4"
format_stamp: "Formatted at 2026-09-24 10:39:54 on dist-test-slave-j38h"
server_key: "c5fe308bc05d08402cb0709fd8cf60fb"
server_key_iv: "94cf6725b5cc2051b20134aeb2b62e6b"
server_key_version: "encryptionkey@0"
I20260924 10:39:54.794692 5478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:54.805706 5478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:54.808884 5478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:54.809098 5478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:54.809679 5478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:54.810477 5478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:54.810518 5478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:54.810556 5478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:54.810613 5478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:54.819296 5478 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:42093
I20260924 10:39:54.819445 5606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:42093 every 8 connection(s)
I20260924 10:39:54.820263 5478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:39:54.827610 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 5478
I20260924 10:39:54.827764 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:39:54.828037 2080 external_mini_cluster.cc:1428] Setting key efd41aa1ea77226a069a5ab5f2e54ad1
Sep 24 10:39:54 dist-test-slave-j38h krb5kdc[5103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246394, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:54.833748 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:54.822295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:42373 (local address 127.2.8.62:43223)
0924 10:39:54.822464 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:54.822469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:54.823220 (+ 751us) server_negotiation.cc:407] Connection header received
0924 10:39:54.824524 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:54.824527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:54.824572 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:54.824646 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:54.825799 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:54.826312 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:54.827416 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:54.827630 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:54.827746 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:54.830257 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:54.830261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:54.830264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:54.830296 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:54.831767 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:54.832258 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:54.832260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:54.832262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:54.832342 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:54.832656 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:54.832658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:54.832660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:54.832849 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:54.832927 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:54.833517 (+ 590us) server_negotiation.cc:299] Negotiation successful
0924 10:39:54.833619 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:39:54.834525 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:54.822634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43223 (local address 127.2.8.3:42373)
0924 10:39:54.823072 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:54.823113 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:54.824338 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:54.824804 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:54.824811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:54.825168 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:54.825669 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:54.825691 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:54.826438 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:54.826440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:54.827300 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:54.827308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:54.827735 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:54.827790 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:54.827969 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:54.828578 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:54.828594 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:54.830092 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:54.831888 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:54.831893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:54.831905 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:54.832163 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:54.832441 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:54.832443 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:54.832444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:54.832573 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:54.832961 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:54.832966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:54.833284 (+ 318us) client_negotiation.cc:769] Sending connection context
0924 10:39:54.833449 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 10:39:54.833700 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
I20260924 10:39:54.835826 5607 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43223
I20260924 10:39:54.836093 5607 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:54.836570 5607 heartbeater.cc:511] Master 127.2.8.62:43223 requested a full tablet report, sending...
I20260924 10:39:54.837601 5148 ts_manager.cc:194] Registered new tserver with Master: 02978715d8dd4732a76623602baec2b4 (127.2.8.3:42093)
I20260924 10:39:54.838162 5148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:42373
I20260924 10:39:54.841835 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 10:39:54 dist-test-slave-j38h krb5kdc[5103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246394, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 10:39:54 dist-test-slave-j38h krb5kdc[5103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246394, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:54.870085 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:54.861493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.2.8.62:43223)
0924 10:39:54.861635 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:54.861640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:54.861781 (+ 141us) server_negotiation.cc:407] Connection header received
0924 10:39:54.861957 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:54.861961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:54.862014 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:54.862099 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:54.862793 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:54.863322 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:54.864157 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:54.864399 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:54.864509 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:54.866899 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:54.866902 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:54.866905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:54.866938 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:54.868394 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:54.868826 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:54.868831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:54.868836 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:54.868915 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:54.869208 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:54.869212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:54.869214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:54.869434 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:54.869570 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:54.869802 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 10:39:54.869917 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 10:39:54.870478 5148 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36458
I20260924 10:39:54.871395 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 5210
I20260924 10:39:54.876452 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 5344
I20260924 10:39:54.881280 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 5478
I20260924 10:39:54.886268 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 5116
2026-09-24T10:39:54Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2134 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:54 dist-test-slave-j38h krb5kdc[5619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:54 dist-test-slave-j38h krb5kdc[5619](info): set up 2 sockets
Sep 24 10:39:54 dist-test-slave-j38h krb5kdc[5619](info): commencing operation
krb5kdc: starting...
W20260924 10:39:55.917083 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:39:56 dist-test-slave-j38h krb5kdc[5619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:39:56.043540 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:33271
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:33271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:56.151116 5632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:56.151371 5632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:56.151410 5632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:56.155385 5632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:56.155445 5632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:56.155465 5632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:56.155481 5632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:56.155496 5632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:56.160636 5632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:33271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:33271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:56.161684 5632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:56.162644 5632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:56.171324 5640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:56.171324 5637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:56.171319 5638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:56.171727 5632 server_base.cc:1034] running on GCE node
I20260924 10:39:56.172240 5632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:56.173352 5632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:56.174549 5632 hybrid_clock.cc:648] HybridClock initialized: now 1790246396174515 us; error 41 us; skew 500 ppm
Sep 24 10:39:56 dist-test-slave-j38h krb5kdc[5619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246396, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:56.177409 5632 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:39:56.178280 5632 webserver.cc:460] Webserver started at http://127.2.8.62:35317/ using document root <none> and password file <none>
I20260924 10:39:56.178797 5632 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:56.178853 5632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:56.179011 5632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:56.180965 5632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "9ab29b5c7ea9478da1111edb7f85de22"
format_stamp: "Formatted at 2026-09-24 10:39:56 on dist-test-slave-j38h"
server_key: "7282a3d2805b837ba20f11525fa9b8e7"
server_key_iv: "186b2fbb7191954ab665607f8f74dab2"
server_key_version: "encryptionkey@0"
I20260924 10:39:56.181571 5632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "9ab29b5c7ea9478da1111edb7f85de22"
format_stamp: "Formatted at 2026-09-24 10:39:56 on dist-test-slave-j38h"
server_key: "7282a3d2805b837ba20f11525fa9b8e7"
server_key_iv: "186b2fbb7191954ab665607f8f74dab2"
server_key_version: "encryptionkey@0"
I20260924 10:39:56.185274 5632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 10:39:56.187541 5647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:56.188702 5632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:56.188797 5632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "9ab29b5c7ea9478da1111edb7f85de22"
format_stamp: "Formatted at 2026-09-24 10:39:56 on dist-test-slave-j38h"
server_key: "7282a3d2805b837ba20f11525fa9b8e7"
server_key_iv: "186b2fbb7191954ab665607f8f74dab2"
server_key_version: "encryptionkey@0"
I20260924 10:39:56.188879 5632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:56.230683 5632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:56.234095 5632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:56.234256 5632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:56.241753 5632 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:33271
I20260924 10:39:56.241773 5699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:33271 every 8 connection(s)
I20260924 10:39:56.242772 5632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:56.245628 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:56.249792 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 5632
I20260924 10:39:56.249913 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:39:56.250216 2080 external_mini_cluster.cc:1428] Setting key 58a889f8aa71a95188253b78758392cd
I20260924 10:39:56.251371 5700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22: Bootstrap starting.
I20260924 10:39:56.253690 5700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:56.254558 5700 log.cc:824] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:56.256681 5700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22: No bootstrap required, opened a new log
Sep 24 10:39:56 dist-test-slave-j38h krb5kdc[5619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:56.260959 5700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab29b5c7ea9478da1111edb7f85de22" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 33271 } }
I20260924 10:39:56.261274 5700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:56.261349 5700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ab29b5c7ea9478da1111edb7f85de22, State: Initialized, Role: FOLLOWER
I20260924 10:39:56.261770 5700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab29b5c7ea9478da1111edb7f85de22" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 33271 } }
I20260924 10:39:56.261912 5700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:56.261977 5700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:56.262080 5700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:56.263188 5700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab29b5c7ea9478da1111edb7f85de22" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 33271 } }
I20260924 10:39:56.263530 5700 leader_election.cc:302] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ab29b5c7ea9478da1111edb7f85de22; no voters:
I20260924 10:39:56.263839 5700 leader_election.cc:288] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:56.264120 5705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:56.264362 5705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [term 1 LEADER]: Becoming Leader. State: Replica: 9ab29b5c7ea9478da1111edb7f85de22, State: Running, Role: LEADER
I20260924 10:39:56.264686 5705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab29b5c7ea9478da1111edb7f85de22" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 33271 } }
I20260924 10:39:56.265193 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:56.251688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.2.8.62:33271)
0924 10:39:56.252071 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:56.252092 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:56.252126 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:39:56.252746 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:56.252763 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:56.253161 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:56.253446 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:56.254354 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:56.255403 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:56.256520 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:56.257096 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:56.257218 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:56.259903 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:56.259908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:56.259924 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:56.259952 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:56.261952 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:56.262460 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:56.262464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:56.262469 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:56.262581 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:56.262881 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:56.262883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:56.262884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:56.263242 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:56.263508 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:56.263865 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 10:39:56.264171 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":248,"thread_start_us":126,"threads_started":1}
I20260924 10:39:56.265304 5700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:56.266331 5707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ab29b5c7ea9478da1111edb7f85de22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab29b5c7ea9478da1111edb7f85de22" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 33271 } } }
I20260924 10:39:56.266469 5707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:56.266705 5707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ab29b5c7ea9478da1111edb7f85de22. Latest consensus state: current_term: 1 leader_uuid: "9ab29b5c7ea9478da1111edb7f85de22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab29b5c7ea9478da1111edb7f85de22" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 33271 } } }
I20260924 10:39:56.266789 5707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22 [sys.catalog]: This master's current role is: LEADER
W20260924 10:39:56.270459 5720 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:39:56.270526 5720 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:39:56.270606 5714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:56.271281 5714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:56.276697 5714 catalog_manager.cc:1347] Generated new cluster ID: c162c1258a1346a7807244fdd427b431
I20260924 10:39:56.276774 5714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:56.296484 5714 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:56.297780 5714 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:56.303548 5714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ab29b5c7ea9478da1111edb7f85de22: Generated new TSK 0
I20260924 10:39:56.304229 5714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:39:56.340503 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:33271
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:56.459688 5726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:56.460013 5726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:56.460095 5726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:56.464210 5726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:56.464311 5726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:56.464406 5726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:39:56.470109 5726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:33271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:56.471177 5726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:56.472198 5726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:56.481657 5734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:56.481791 5726 server_base.cc:1034] running on GCE node
W20260924 10:39:56.483465 5731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:56.483661 5732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:56.484107 5726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:56.484774 5726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:56.485971 5726 hybrid_clock.cc:648] HybridClock initialized: now 1790246396485941 us; error 39 us; skew 500 ppm
Sep 24 10:39:56 dist-test-slave-j38h krb5kdc[5619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246396, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:56.488796 5726 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:39:56.489766 5726 webserver.cc:460] Webserver started at http://127.2.8.1:36373/ using document root <none> and password file <none>
I20260924 10:39:56.490329 5726 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:56.490456 5726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:56.490692 5726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:56.492827 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "903f376469c74615b77e8acc77ac7635"
format_stamp: "Formatted at 2026-09-24 10:39:56 on dist-test-slave-j38h"
server_key: "64bf12fcee613efe1d2b8fc24c15a872"
server_key_iv: "bce1f728aad3052adc535d3fa8a5e83e"
server_key_version: "encryptionkey@0"
I20260924 10:39:56.493449 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "903f376469c74615b77e8acc77ac7635"
format_stamp: "Formatted at 2026-09-24 10:39:56 on dist-test-slave-j38h"
server_key: "64bf12fcee613efe1d2b8fc24c15a872"
server_key_iv: "bce1f728aad3052adc535d3fa8a5e83e"
server_key_version: "encryptionkey@0"
I20260924 10:39:56.497184 5726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:39:56.499544 5741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:56.500838 5726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:56.500968 5726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "903f376469c74615b77e8acc77ac7635"
format_stamp: "Formatted at 2026-09-24 10:39:56 on dist-test-slave-j38h"
server_key: "64bf12fcee613efe1d2b8fc24c15a872"
server_key_iv: "bce1f728aad3052adc535d3fa8a5e83e"
server_key_version: "encryptionkey@0"
I20260924 10:39:56.501135 5726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:56.516151 5726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:56.519105 5726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:56.519299 5726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:56.519873 5726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:56.520663 5726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:56.520732 5726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:56.520802 5726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:56.520854 5726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:56.530050 5726 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:46595
I20260924 10:39:56.530071 5854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:46595 every 8 connection(s)
I20260924 10:39:56.530900 5726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:39:56.537227 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 5726
I20260924 10:39:56.537372 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:39:56.537638 2080 external_mini_cluster.cc:1428] Setting key 4e9538d6c44b14d43701a5e8663f8258
Sep 24 10:39:56 dist-test-slave-j38h krb5kdc[5619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246396, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:56.544581 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:56.532637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:60645 (local address 127.2.8.62:33271)
0924 10:39:56.532793 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:56.532798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:56.533639 (+ 841us) server_negotiation.cc:407] Connection header received
0924 10:39:56.534945 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:56.534948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:56.534993 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:56.535066 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:56.536157 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:56.536670 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:56.537910 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:56.538179 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:56.538262 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:56.540848 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:56.540851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:56.540853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:56.540876 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:56.542361 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:56.543006 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:56.543009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:56.543010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:56.543080 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:56.543490 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:56.543492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:56.543493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:56.543648 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:56.543719 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:56.544366 (+ 647us) server_negotiation.cc:299] Negotiation successful
0924 10:39:56.544465 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:39:56.545806 5857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:56.533074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:33271 (local address 127.2.8.1:60645)
0924 10:39:56.533486 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:56.533527 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:56.534744 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:56.535223 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:56.535229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:56.535578 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:56.536036 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:56.536049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:56.536773 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:56.536775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:56.537798 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:56.537807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:56.538278 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:56.538324 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:56.538489 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:56.539099 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:56.539114 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:56.540671 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:56.542511 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:56.542520 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:56.542540 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:56.542900 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:56.543177 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:56.543183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:56.543188 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:56.543378 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:56.543754 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:56.543763 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:56.544104 (+ 341us) client_negotiation.cc:769] Sending connection context
0924 10:39:56.544329 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 10:39:56.544689 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
I20260924 10:39:56.547061 5855 heartbeater.cc:347] Connected to a master server at 127.2.8.62:33271
I20260924 10:39:56.547286 5855 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:56.547712 5855 heartbeater.cc:511] Master 127.2.8.62:33271 requested a full tablet report, sending...
I20260924 10:39:56.549218 5664 ts_manager.cc:194] Registered new tserver with Master: 903f376469c74615b77e8acc77ac7635 (127.2.8.1:46595)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
I20260924 10:39:56.550805 5664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:60645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:39:56.563843 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:33271
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:56.673110 5860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:56.673403 5860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:56.673480 5860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:56.677419 5860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:56.677505 5860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:56.677636 5860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:39:56.682757 5860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:33271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:56.683775 5860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:56.684772 5860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:56.693975 5866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:56.694042 5865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:56.693980 5868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:56.694366 5860 server_base.cc:1034] running on GCE node
I20260924 10:39:56.694811 5860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:56.695468 5860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:56.696647 5860 hybrid_clock.cc:648] HybridClock initialized: now 1790246396696634 us; error 25 us; skew 500 ppm
Sep 24 10:39:56 dist-test-slave-j38h krb5kdc[5619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246396, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:56.699354 5860 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:39:56.700196 5860 webserver.cc:460] Webserver started at http://127.2.8.2:45163/ using document root <none> and password file <none>
I20260924 10:39:56.700841 5860 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:56.700941 5860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:56.701184 5860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:56.703073 5860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "414703e4c8fa426fad18f0647032c780"
format_stamp: "Formatted at 2026-09-24 10:39:56 on dist-test-slave-j38h"
server_key: "d00c2932e4220104a278606e24e544f9"
server_key_iv: "f9f382ec2631722b82ceac7fbd95ca4e"
server_key_version: "encryptionkey@0"
I20260924 10:39:56.703545 5860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "414703e4c8fa426fad18f0647032c780"
format_stamp: "Formatted at 2026-09-24 10:39:56 on dist-test-slave-j38h"
server_key: "d00c2932e4220104a278606e24e544f9"
server_key_iv: "f9f382ec2631722b82ceac7fbd95ca4e"
server_key_version: "encryptionkey@0"
I20260924 10:39:56.707082 5860 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:39:56.709300 5875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:56.710445 5860 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:56.710532 5860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "414703e4c8fa426fad18f0647032c780"
format_stamp: "Formatted at 2026-09-24 10:39:56 on dist-test-slave-j38h"
server_key: "d00c2932e4220104a278606e24e544f9"
server_key_iv: "f9f382ec2631722b82ceac7fbd95ca4e"
server_key_version: "encryptionkey@0"
I20260924 10:39:56.710610 5860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:56.738184 5860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:56.741114 5860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:56.741263 5860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:56.741775 5860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:56.742529 5860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:56.742574 5860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:56.742612 5860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:56.742627 5860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:56.751396 5860 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:45703
I20260924 10:39:56.751426 5988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:45703 every 8 connection(s)
I20260924 10:39:56.752234 5860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:39:56.760895 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 5860
Sep 24 10:39:56 dist-test-slave-j38h krb5kdc[5619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246396, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:56.761050 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:39:56.761348 2080 external_mini_cluster.cc:1428] Setting key fa260318ce082b2e88524a440ecf6ed3
I20260924 10:39:56.765504 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:56.753965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:37567 (local address 127.2.8.62:33271)
0924 10:39:56.754123 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:56.754127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:56.754854 (+ 727us) server_negotiation.cc:407] Connection header received
0924 10:39:56.756256 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:56.756258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:56.756304 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:56.756378 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:56.757499 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:56.758020 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:56.759062 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:56.759256 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:56.759326 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:56.761718 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:56.761721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:56.761723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:56.761744 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:56.763160 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:56.763664 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:56.763669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:56.763671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:56.763753 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:56.764156 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:56.764162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:56.764165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:56.764402 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:56.764485 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:56.765220 (+ 735us) server_negotiation.cc:299] Negotiation successful
0924 10:39:56.765332 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:39:56.766433 5991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:56.754262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:33271 (local address 127.2.8.2:37567)
0924 10:39:56.754686 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:56.754726 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:56.756057 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:56.756544 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:56.756551 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:56.756890 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:56.757377 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:56.757391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:56.758119 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:56.758121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:56.758967 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:56.758975 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:56.759333 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:56.759376 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:56.759506 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:56.760031 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:56.760046 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:56.761556 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:56.763275 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:56.763281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:56.763295 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:56.763551 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:56.763852 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:56.763855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:56.763857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:56.763999 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:56.764579 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:56.764588 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:56.764914 (+ 326us) client_negotiation.cc:769] Sending connection context
0924 10:39:56.765178 (+ 264us) client_negotiation.cc:240] Negotiation successful
0924 10:39:56.765485 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
I20260924 10:39:56.767763 5989 heartbeater.cc:347] Connected to a master server at 127.2.8.62:33271
I20260924 10:39:56.768018 5989 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:56.768628 5989 heartbeater.cc:511] Master 127.2.8.62:33271 requested a full tablet report, sending...
I20260924 10:39:56.769762 5664 ts_manager.cc:194] Registered new tserver with Master: 414703e4c8fa426fad18f0647032c780 (127.2.8.2:45703)
I20260924 10:39:56.770610 5664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:37567
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:39:56.788512 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:33271
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:56.902717 5994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:56.902963 5994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:56.903004 5994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:56.907092 5994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:56.907148 5994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:56.907207 5994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:39:56.912343 5994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:33271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:56.913394 5994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:56.914461 5994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:56.923796 6000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:56.923796 6002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:56.923914 5994 server_base.cc:1034] running on GCE node
W20260924 10:39:56.923816 5999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:56.924502 5994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:56.925149 5994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:56.926366 5994 hybrid_clock.cc:648] HybridClock initialized: now 1790246396926337 us; error 40 us; skew 500 ppm
Sep 24 10:39:56 dist-test-slave-j38h krb5kdc[5619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246396, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:56.929503 5994 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:39:56.930451 5994 webserver.cc:460] Webserver started at http://127.2.8.3:44469/ using document root <none> and password file <none>
I20260924 10:39:56.930989 5994 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:56.931051 5994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:56.931218 5994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:56.933439 5994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "5d02d42d45f54c419aa8bb3fa3f19e2f"
format_stamp: "Formatted at 2026-09-24 10:39:56 on dist-test-slave-j38h"
server_key: "acd0a5237bad50dc5090f4bba97cb639"
server_key_iv: "3fe0012efaac7571c7edcdf56c647738"
server_key_version: "encryptionkey@0"
I20260924 10:39:56.934044 5994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "5d02d42d45f54c419aa8bb3fa3f19e2f"
format_stamp: "Formatted at 2026-09-24 10:39:56 on dist-test-slave-j38h"
server_key: "acd0a5237bad50dc5090f4bba97cb639"
server_key_iv: "3fe0012efaac7571c7edcdf56c647738"
server_key_version: "encryptionkey@0"
I20260924 10:39:56.937964 5994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:39:56.940507 6009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:56.941902 5994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:39:56.942034 5994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "5d02d42d45f54c419aa8bb3fa3f19e2f"
format_stamp: "Formatted at 2026-09-24 10:39:56 on dist-test-slave-j38h"
server_key: "acd0a5237bad50dc5090f4bba97cb639"
server_key_iv: "3fe0012efaac7571c7edcdf56c647738"
server_key_version: "encryptionkey@0"
I20260924 10:39:56.942152 5994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:56.954916 5994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:56.957962 5994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:56.958173 5994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:56.958753 5994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:56.959594 5994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:56.959645 5994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:56.959717 5994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:56.959758 5994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:56.969941 5994 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:36163
I20260924 10:39:56.969959 6122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:36163 every 8 connection(s)
I20260924 10:39:56.970903 5994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:39:56.974668 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 5994
I20260924 10:39:56.974812 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:39:56.975096 2080 external_mini_cluster.cc:1428] Setting key 86fa8f0951877af67abade9183569c13
Sep 24 10:39:56 dist-test-slave-j38h krb5kdc[5619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246396, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:56.984746 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:56.972659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:58855 (local address 127.2.8.62:33271)
0924 10:39:56.972820 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:56.972824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:56.973654 (+ 830us) server_negotiation.cc:407] Connection header received
0924 10:39:56.975305 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:56.975308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:56.975370 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:56.975484 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:56.976927 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:56.977600 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:56.978788 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:56.978989 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:56.979063 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:56.981478 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:56.981481 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:56.981483 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:56.981507 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:56.982912 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:56.983479 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:56.983482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:56.983483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:56.983552 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:56.983829 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:56.983831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:56.983832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:56.983985 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:56.984057 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:56.984547 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 10:39:56.984631 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 10:39:56.985621 6125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:56.973070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:33271 (local address 127.2.8.3:58855)
0924 10:39:56.973496 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:56.973540 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:56.975074 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:56.975673 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:56.975680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:56.976131 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:56.976767 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:56.976796 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:56.977756 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:56.977759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:56.978681 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:56.978690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:56.979069 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:56.979110 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:56.979246 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:56.979752 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:56.979768 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:56.981274 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:56.983064 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:56.983069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:56.983079 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:56.983378 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:56.983627 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:56.983628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:56.983630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:56.983751 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:56.984068 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:56.984072 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:56.984336 (+ 264us) client_negotiation.cc:769] Sending connection context
0924 10:39:56.984504 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 10:39:56.984738 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":96,"threads_started":1}
I20260924 10:39:56.986763 6123 heartbeater.cc:347] Connected to a master server at 127.2.8.62:33271
I20260924 10:39:56.986976 6123 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:56.987404 6123 heartbeater.cc:511] Master 127.2.8.62:33271 requested a full tablet report, sending...
I20260924 10:39:56.988479 5664 ts_manager.cc:194] Registered new tserver with Master: 5d02d42d45f54c419aa8bb3fa3f19e2f (127.2.8.3:36163)
I20260924 10:39:56.989183 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:39:56.989250 5664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:58855
Sep 24 10:39:56 dist-test-slave-j38h krb5kdc[5619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:39:57 dist-test-slave-j38h krb5kdc[5619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.1@KRBTEST.COM
I20260924 10:39:57.014381 6130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:57.005509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.2.8.1:46595)
0924 10:39:57.005801 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:57.005815 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:57.005843 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:39:57.005904 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:57.005910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:57.006109 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:57.006253 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:57.006928 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:57.007685 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:57.008520 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:57.008879 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:57.009074 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:57.011321 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:57.011328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:57.011340 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:57.011362 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:57.012743 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:57.013198 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:57.013202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:57.013204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:57.013277 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:57.013502 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:57.013504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:57.013505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:57.013718 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:57.013880 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:57.014030 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 10:39:57.014222 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
Sep 24 10:39:57 dist-test-slave-j38h krb5kdc[5619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:57.024451 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:57.017199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.2.8.62:33271)
0924 10:39:57.017371 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:57.017375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:57.017426 (+ 51us) server_negotiation.cc:407] Connection header received
0924 10:39:57.017510 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:57.017512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:57.017554 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:57.017614 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:57.018099 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:57.018589 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:57.019276 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:57.019480 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:57.019551 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:57.021655 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:57.021658 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:57.021660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:57.021682 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:57.023076 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:57.023410 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:57.023413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:57.023414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:57.023496 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:57.023701 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:57.023703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:57.023704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:57.023889 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:57.023990 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:57.024136 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 10:39:57.024222 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 10:39:57.024818 5664 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55072
I20260924 10:39:57.025501 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 5726
I20260924 10:39:57.030712 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 5860
I20260924 10:39:57.035583 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 5994
I20260924 10:39:57.040436 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 5632
2026-09-24T10:39:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2154 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:39:57 dist-test-slave-j38h krb5kdc[6135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:39:57 dist-test-slave-j38h krb5kdc[6135](info): set up 2 sockets
Sep 24 10:39:57 dist-test-slave-j38h krb5kdc[6135](info): commencing operation
krb5kdc: starting...
W20260924 10:39:58.185145 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:39:58 dist-test-slave-j38h krb5kdc[6135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:39:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:39:58.304544 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:35893
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:35893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:58.409821 6148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:58.410111 6148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:58.410182 6148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:58.414186 6148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:58.414263 6148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:58.414304 6148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:58.414359 6148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:58.414397 6148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:58.419454 6148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:35893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:35893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:58.420470 6148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:58.421512 6148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:58.429402 6154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:58.429461 6156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:58.429598 6148 server_base.cc:1034] running on GCE node
W20260924 10:39:58.429415 6153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:58.430243 6148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:58.431277 6148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:58.432456 6148 hybrid_clock.cc:648] HybridClock initialized: now 1790246398432436 us; error 30 us; skew 500 ppm
Sep 24 10:39:58 dist-test-slave-j38h krb5kdc[6135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246398, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:58.439494 6148 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:39:58.440220 6148 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:58.440284 6148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:58.440543 6148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:58.443058 6148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "71df6df6f05041d2a71df7255e29a035"
format_stamp: "Formatted at 2026-09-24 10:39:58 on dist-test-slave-j38h"
server_key: "8d8bf949284805a0aab945c8da5cb441"
server_key_iv: "86f56cfc56adc4ee30eac1bf8536de53"
server_key_version: "encryptionkey@0"
I20260924 10:39:58.443615 6148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "71df6df6f05041d2a71df7255e29a035"
format_stamp: "Formatted at 2026-09-24 10:39:58 on dist-test-slave-j38h"
server_key: "8d8bf949284805a0aab945c8da5cb441"
server_key_iv: "86f56cfc56adc4ee30eac1bf8536de53"
server_key_version: "encryptionkey@0"
I20260924 10:39:58.447165 6148 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20260924 10:39:58.449518 6162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:58.450757 6148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:39:58.450903 6148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "71df6df6f05041d2a71df7255e29a035"
format_stamp: "Formatted at 2026-09-24 10:39:58 on dist-test-slave-j38h"
server_key: "8d8bf949284805a0aab945c8da5cb441"
server_key_iv: "86f56cfc56adc4ee30eac1bf8536de53"
server_key_version: "encryptionkey@0"
I20260924 10:39:58.451009 6148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:58.462530 6148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:58.465346 6148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:58.465545 6148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:58.472771 6148 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:35893
I20260924 10:39:58.473096 6214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:35893 every 8 connection(s)
I20260924 10:39:58.473486 6148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:58.476130 6215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:58.480137 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 6148
I20260924 10:39:58.480286 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:39:58.480551 2080 external_mini_cluster.cc:1428] Setting key a7a1d36302622f8a80936fe2f0769e6b
I20260924 10:39:58.481886 6215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035: Bootstrap starting.
I20260924 10:39:58.484170 6215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:58.484906 6215 log.cc:824] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:58.486639 6215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035: No bootstrap required, opened a new log
Sep 24 10:39:58 dist-test-slave-j38h krb5kdc[6135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:58.490676 6215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71df6df6f05041d2a71df7255e29a035" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 35893 } }
I20260924 10:39:58.490964 6215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:58.491021 6215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71df6df6f05041d2a71df7255e29a035, State: Initialized, Role: FOLLOWER
I20260924 10:39:58.491506 6215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71df6df6f05041d2a71df7255e29a035" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 35893 } }
I20260924 10:39:58.491663 6215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:58.491719 6215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:58.491822 6215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:58.492899 6215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71df6df6f05041d2a71df7255e29a035" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 35893 } }
I20260924 10:39:58.493250 6215 leader_election.cc:302] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71df6df6f05041d2a71df7255e29a035; no voters:
I20260924 10:39:58.493604 6215 leader_election.cc:288] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:58.493705 6220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:58.493973 6220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [term 1 LEADER]: Becoming Leader. State: Replica: 71df6df6f05041d2a71df7255e29a035, State: Running, Role: LEADER
I20260924 10:39:58.494321 6220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71df6df6f05041d2a71df7255e29a035" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 35893 } }
I20260924 10:39:58.494828 6215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:58.494992 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:58.481888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.2.8.62:35893)
0924 10:39:58.482284 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:58.482302 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:58.482344 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:39:58.482957 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:58.482974 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:58.483287 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:58.483554 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:58.484453 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:58.485521 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:58.486346 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:58.486847 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:58.487010 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:58.489716 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:58.489722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:58.489741 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:58.489770 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:58.491716 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:58.492240 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:58.492245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:58.492251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:58.492376 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:58.492640 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:58.492643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:58.492646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:58.493186 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:58.493371 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:58.493696 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 10:39:58.494003 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":120,"threads_started":1}
I20260924 10:39:58.496775 6221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71df6df6f05041d2a71df7255e29a035. Latest consensus state: current_term: 1 leader_uuid: "71df6df6f05041d2a71df7255e29a035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71df6df6f05041d2a71df7255e29a035" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 35893 } } }
I20260924 10:39:58.496896 6221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:58.497243 6235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:58.497443 6222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71df6df6f05041d2a71df7255e29a035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71df6df6f05041d2a71df7255e29a035" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 35893 } } }
I20260924 10:39:58.497618 6222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035 [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:58.500344 6235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:58.505625 6235 catalog_manager.cc:1347] Generated new cluster ID: a7f3d1de67df4cefb68e878547e40375
I20260924 10:39:58.505697 6235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:58.516402 6235 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:58.517359 6235 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:58.524051 6235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035: Generated new TSK 0
I20260924 10:39:58.524696 6235 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:39:58.552357 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:35893
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:58.663388 6241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:58.663621 6241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:58.663663 6241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:58.667610 6241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:58.667661 6241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:58.667718 6241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:39:58.672721 6241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:35893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:58.673702 6241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:58.674628 6241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:58.683753 6246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:58.683746 6247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:58.683808 6249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:58.684106 6241 server_base.cc:1034] running on GCE node
I20260924 10:39:58.684580 6241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:58.685308 6241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:58.686506 6241 hybrid_clock.cc:648] HybridClock initialized: now 1790246398686489 us; error 25 us; skew 500 ppm
Sep 24 10:39:58 dist-test-slave-j38h krb5kdc[6135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246398, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:58.689465 6241 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:39:58.690189 6241 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:58.690253 6241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:58.690474 6241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:58.692481 6241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "16167dd7162549ee9a03278d53fbbb3b"
format_stamp: "Formatted at 2026-09-24 10:39:58 on dist-test-slave-j38h"
server_key: "896da38b4eb760bc070d19e584c8e79d"
server_key_iv: "28e8dd0e3bf12e1880ed231909817148"
server_key_version: "encryptionkey@0"
I20260924 10:39:58.693100 6241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "16167dd7162549ee9a03278d53fbbb3b"
format_stamp: "Formatted at 2026-09-24 10:39:58 on dist-test-slave-j38h"
server_key: "896da38b4eb760bc070d19e584c8e79d"
server_key_iv: "28e8dd0e3bf12e1880ed231909817148"
server_key_version: "encryptionkey@0"
I20260924 10:39:58.697057 6241 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 10:39:58.699457 6255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:58.700773 6241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:58.700922 6241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "16167dd7162549ee9a03278d53fbbb3b"
format_stamp: "Formatted at 2026-09-24 10:39:58 on dist-test-slave-j38h"
server_key: "896da38b4eb760bc070d19e584c8e79d"
server_key_iv: "28e8dd0e3bf12e1880ed231909817148"
server_key_version: "encryptionkey@0"
I20260924 10:39:58.701086 6241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:58.711898 6241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:58.714607 6241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:58.714795 6241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:58.715320 6241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:58.716081 6241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:58.716125 6241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:58.716187 6241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:58.716220 6241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:58.724594 6241 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:37479
I20260924 10:39:58.724620 6368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:37479 every 8 connection(s)
I20260924 10:39:58.725142 6241 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:39:58.728588 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 6241
I20260924 10:39:58.728727 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:39:58.729038 2080 external_mini_cluster.cc:1428] Setting key a34789a1649d4a962d2733cfaee2cdb7
Sep 24 10:39:58 dist-test-slave-j38h krb5kdc[6135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246398, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
I20260924 10:39:58.738946 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:58.726770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:54041 (local address 127.2.8.62:35893)
0924 10:39:58.726898 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:58.726902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:58.727757 (+ 855us) server_negotiation.cc:407] Connection header received
0924 10:39:58.729245 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:58.729248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:58.729307 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:58.729382 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:58.730680 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:58.731207 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:58.732321 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:58.732551 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:58.732662 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:58.735274 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:58.735277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:58.735279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:58.735301 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:58.736734 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:58.737417 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:58.737420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:58.737422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:58.737511 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:58.737865 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:58.737868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:58.737870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:58.738078 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:58.738183 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:58.738653 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 10:39:58.738769 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:39:58.739718 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:58.727140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:35893 (local address 127.2.8.1:54041)
0924 10:39:58.727612 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:58.727650 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:58.729017 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:58.729617 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:58.729626 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:58.729998 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:58.730545 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:58.730558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:58.731329 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:58.731333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:58.732197 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:58.732206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:58.732648 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:58.732696 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:58.732864 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:58.733492 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:58.733507 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:58.735109 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:58.736918 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:58.736924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:58.736946 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:58.737251 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:58.737611 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:58.737615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:58.737617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:58.737758 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:58.738181 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:58.738187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:58.738421 (+ 234us) client_negotiation.cc:769] Sending connection context
0924 10:39:58.738599 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 10:39:58.738898 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
I20260924 10:39:58.741168 6369 heartbeater.cc:347] Connected to a master server at 127.2.8.62:35893
I20260924 10:39:58.741405 6369 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:58.741824 6369 heartbeater.cc:511] Master 127.2.8.62:35893 requested a full tablet report, sending...
I20260924 10:39:58.743403 6179 ts_manager.cc:194] Registered new tserver with Master: 16167dd7162549ee9a03278d53fbbb3b (127.2.8.1:37479)
I20260924 10:39:58.745079 6179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:54041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:39:58.756110 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:35893
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:58.868300 6374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:58.868562 6374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:58.868643 6374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:58.872840 6374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:58.872895 6374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:58.873019 6374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:39:58.878340 6374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:35893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:58.879422 6374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:58.880473 6374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:58.890082 6382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:58.890196 6374 server_base.cc:1034] running on GCE node
W20260924 10:39:58.890100 6380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:58.890403 6379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:58.890916 6374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:58.891579 6374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:58.892782 6374 hybrid_clock.cc:648] HybridClock initialized: now 1790246398892759 us; error 25 us; skew 500 ppm
Sep 24 10:39:58 dist-test-slave-j38h krb5kdc[6135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246398, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:58.895920 6374 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:39:58.896732 6374 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:58.896834 6374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:58.897101 6374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:58.899592 6374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "5297c88fc608471d9ded8df50d3fc3e8"
format_stamp: "Formatted at 2026-09-24 10:39:58 on dist-test-slave-j38h"
server_key: "9f40cbfd0b6914d797c493cf05a2c564"
server_key_iv: "ce06b5c5a6ef21ac200451a827bb447c"
server_key_version: "encryptionkey@0"
I20260924 10:39:58.900216 6374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "5297c88fc608471d9ded8df50d3fc3e8"
format_stamp: "Formatted at 2026-09-24 10:39:58 on dist-test-slave-j38h"
server_key: "9f40cbfd0b6914d797c493cf05a2c564"
server_key_iv: "ce06b5c5a6ef21ac200451a827bb447c"
server_key_version: "encryptionkey@0"
I20260924 10:39:58.903947 6374 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 10:39:58.906438 6388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:58.907801 6374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:39:58.907956 6374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "5297c88fc608471d9ded8df50d3fc3e8"
format_stamp: "Formatted at 2026-09-24 10:39:58 on dist-test-slave-j38h"
server_key: "9f40cbfd0b6914d797c493cf05a2c564"
server_key_iv: "ce06b5c5a6ef21ac200451a827bb447c"
server_key_version: "encryptionkey@0"
I20260924 10:39:58.908066 6374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:58.932171 6374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:58.935017 6374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:58.935222 6374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:58.935813 6374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:58.936627 6374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:58.936698 6374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:58.936769 6374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:58.936822 6374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:58.945823 6374 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:41457
I20260924 10:39:58.945892 6501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:41457 every 8 connection(s)
I20260924 10:39:58.946436 6374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:39:58.952486 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 6374
I20260924 10:39:58.952625 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:39:58.952996 2080 external_mini_cluster.cc:1428] Setting key b56ae1d721433efdbdeeb9e52f88ef4e
Sep 24 10:39:58 dist-test-slave-j38h krb5kdc[6135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246398, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:58.961455 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:58.948164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:60919 (local address 127.2.8.62:35893)
0924 10:39:58.948310 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:58.948315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:58.949281 (+ 966us) server_negotiation.cc:407] Connection header received
0924 10:39:58.950725 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:58.950728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:58.950772 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:58.950845 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:58.951983 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:58.952495 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:58.954168 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:58.954500 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:58.954632 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:58.957873 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:58.957877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:58.957879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:58.957903 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:58.959343 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:58.959898 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:58.959903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:58.959904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:58.959984 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:58.960332 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:58.960336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:58.960337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:58.960524 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:58.960630 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:58.961157 (+ 527us) server_negotiation.cc:299] Negotiation successful
0924 10:39:58.961275 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:39:58.962227 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:58.948563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:35893 (local address 127.2.8.2:60919)
0924 10:39:58.949121 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:58.949176 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:58.950530 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:58.950981 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:58.950988 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:58.951364 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:58.951858 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:58.951872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:58.952661 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:58.952667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:58.954042 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:58.954051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:58.954610 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:58.954666 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:58.954902 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:58.955551 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:58.955578 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:58.957680 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:58.959510 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:58.959515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:58.959532 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:58.959787 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:58.960093 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:58.960096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:58.960098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:58.960231 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:58.960628 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:58.960635 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:58.960874 (+ 239us) client_negotiation.cc:769] Sending connection context
0924 10:39:58.961100 (+ 226us) client_negotiation.cc:240] Negotiation successful
0924 10:39:58.961415 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
I20260924 10:39:58.963368 6502 heartbeater.cc:347] Connected to a master server at 127.2.8.62:35893
I20260924 10:39:58.963620 6502 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:58.964056 6502 heartbeater.cc:511] Master 127.2.8.62:35893 requested a full tablet report, sending...
I20260924 10:39:58.965097 6179 ts_manager.cc:194] Registered new tserver with Master: 5297c88fc608471d9ded8df50d3fc3e8 (127.2.8.2:41457)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
I20260924 10:39:58.965859 6179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:60919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:39:58.979806 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:35893
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:39:59.087626 6507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:59.087863 6507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:59.087908 6507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:59.091966 6507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:59.092021 6507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:59.092134 6507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:39:59.097388 6507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:35893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:59.098337 6507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:59.099283 6507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:59.108827 6513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:59.108805 6512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:59.108850 6515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:59.109108 6507 server_base.cc:1034] running on GCE node
I20260924 10:39:59.109558 6507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:59.110159 6507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:59.111315 6507 hybrid_clock.cc:648] HybridClock initialized: now 1790246399111300 us; error 27 us; skew 500 ppm
Sep 24 10:39:59 dist-test-slave-j38h krb5kdc[6135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246399, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:39:59.114081 6507 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:39:59.114784 6507 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:59.114843 6507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:59.115046 6507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:59.117065 6507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "67fc7010bb754a44abacbc6ff9f911de"
format_stamp: "Formatted at 2026-09-24 10:39:59 on dist-test-slave-j38h"
server_key: "8ed5a35581a081f028a075808e5a6d31"
server_key_iv: "2c2c6c0000a51f6a51a459f3b6b8ed77"
server_key_version: "encryptionkey@0"
I20260924 10:39:59.117628 6507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "67fc7010bb754a44abacbc6ff9f911de"
format_stamp: "Formatted at 2026-09-24 10:39:59 on dist-test-slave-j38h"
server_key: "8ed5a35581a081f028a075808e5a6d31"
server_key_iv: "2c2c6c0000a51f6a51a459f3b6b8ed77"
server_key_version: "encryptionkey@0"
I20260924 10:39:59.121156 6507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:39:59.123423 6521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:59.124600 6507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:59.124723 6507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "67fc7010bb754a44abacbc6ff9f911de"
format_stamp: "Formatted at 2026-09-24 10:39:59 on dist-test-slave-j38h"
server_key: "8ed5a35581a081f028a075808e5a6d31"
server_key_iv: "2c2c6c0000a51f6a51a459f3b6b8ed77"
server_key_version: "encryptionkey@0"
I20260924 10:39:59.124825 6507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:59.141853 6507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:59.144971 6507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:59.145203 6507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:59.145777 6507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:59.146592 6507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:59.146659 6507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:59.146727 6507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:59.146776 6507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:59.155889 6507 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:43599
I20260924 10:39:59.155907 6634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:43599 every 8 connection(s)
I20260924 10:39:59.156463 6507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:39:59.165717 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 6507
I20260924 10:39:59.165833 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:39:59.166088 2080 external_mini_cluster.cc:1428] Setting key a4ff897fab8aabda028a5faaa470471b
Sep 24 10:39:59 dist-test-slave-j38h krb5kdc[6135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246399, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:39:59.170389 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.158393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:45555 (local address 127.2.8.62:35893)
0924 10:39:59.158583 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:59.158587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:59.159449 (+ 862us) server_negotiation.cc:407] Connection header received
0924 10:39:59.160858 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:59.160861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:59.160919 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:59.161036 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:59.162184 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.162716 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.163862 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.164092 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.164202 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:59.166734 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:59.166738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:59.166740 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:59.166762 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:59.168479 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:59.168916 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:59.168918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:59.168919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:59.169033 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:59.169367 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:59.169371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:59.169372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:59.169578 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:59.169670 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:59.170098 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 10:39:59.170211 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 10:39:59.171093 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.158815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:35893 (local address 127.2.8.3:45555)
0924 10:39:59.159247 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:59.159335 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:59.160623 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:59.161211 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:59.161218 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:59.161581 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:59.162033 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:59.162047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.162865 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.162873 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:59.163734 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:59.163743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.164188 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.164237 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:59.164411 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:59.164906 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:39:59.164921 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:59.166590 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:59.168575 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:59.168580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:59.168591 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:59.168833 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:59.169121 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:39:59.169123 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:39:59.169124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:39:59.169250 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:39:59.169672 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:59.169677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:59.169883 (+ 206us) client_negotiation.cc:769] Sending connection context
0924 10:39:59.170038 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 10:39:59.170265 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260924 10:39:59.172259 6635 heartbeater.cc:347] Connected to a master server at 127.2.8.62:35893
I20260924 10:39:59.172479 6635 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:59.172904 6635 heartbeater.cc:511] Master 127.2.8.62:35893 requested a full tablet report, sending...
I20260924 10:39:59.173920 6179 ts_manager.cc:194] Registered new tserver with Master: 67fc7010bb754a44abacbc6ff9f911de (127.2.8.3:43599)
I20260924 10:39:59.174467 6179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:45555
I20260924 10:39:59.179979 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:39:59.188931 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.182121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59132 (local address 127.2.8.62:35893)
0924 10:39:59.182257 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:59.182261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:59.182379 (+ 118us) server_negotiation.cc:407] Connection header received
0924 10:39:59.182487 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:59.182490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:59.182524 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:59.182582 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:59.183239 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.183717 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.184597 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.184792 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.184862 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:59.186092 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:59.186094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:59.186096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:39:59.186116 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:59.187519 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:59.187915 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:59.187917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:59.187918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:59.187988 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:39:59.188213 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:39:59.188215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:39:59.188216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:39:59.188397 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:59.188510 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:59.188668 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 10:39:59.188754 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:39:59.191620 6179 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:39:59.194680 6179 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:39:59.209609 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.203435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:43599 (local address 127.0.0.1:37456)
0924 10:39:59.204533 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:59.204548 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:59.204662 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:59.205414 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:59.205417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:59.205446 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:59.205646 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:59.205653 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.207055 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.207059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:59.208120 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:59.208130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.209029 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.209033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:59.209145 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:59.209185 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 10:39:59.209303 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 10:39:59.209426 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"mutex_wait_us":104,"spinlock_wait_cycles":164352,"thread_start_us":109,"threads_started":1}
I20260924 10:39:59.209793 6648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.203480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.2.8.2:41457)
0924 10:39:59.203911 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:59.203924 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:59.204438 (+ 514us) server_negotiation.cc:407] Connection header received
0924 10:39:59.204695 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:59.204702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:59.204915 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:59.205107 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:59.205394 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.206561 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.208328 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.209149 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.209280 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:59.209421 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:59.209531 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 10:39:59.209627 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":103,"threads_started":1}
I20260924 10:39:59.209955 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.204069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.2.8.3:43599)
0924 10:39:59.204843 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:59.204853 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:59.204873 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:39:59.204938 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:59.204944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:59.205130 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:59.205262 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:59.205781 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.206862 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.208256 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.208867 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.209017 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:59.209172 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:59.209695 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 10:39:59.209807 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":153,"threads_started":1}
I20260924 10:39:59.210270 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.203299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:41457 (local address 127.0.0.1:44970)
0924 10:39:59.204300 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:59.204352 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:59.204481 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:59.205118 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:59.205124 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:59.205146 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:59.205277 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:59.205283 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.206706 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.206709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:59.208197 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:59.208207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.209285 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.209292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:59.209393 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:59.209413 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:39:59.209496 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 10:39:59.209557 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":78,"thread_start_us":78,"threads_started":1}
I20260924 10:39:59.210362 6651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.204124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45908 (local address 127.2.8.1:37479)
0924 10:39:59.204672 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:59.204679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:59.204918 (+ 239us) server_negotiation.cc:407] Connection header received
0924 10:39:59.205098 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:59.205104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:59.205297 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:59.205454 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:59.205846 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.206929 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.208496 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.209284 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.209371 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:59.209498 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:59.210110 (+ 612us) server_negotiation.cc:299] Negotiation successful
0924 10:39:59.210211 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":103,"threads_started":1}
I20260924 10:39:59.210659 6647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.203653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:37479 (local address 127.0.0.1:45908)
0924 10:39:59.204822 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:59.204832 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:59.204934 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:59.205437 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:59.205440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:59.205535 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:59.205661 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:59.205667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.207077 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.207080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:59.208367 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:59.208374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.209556 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.209560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:59.209649 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:59.209671 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:39:59.209741 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 10:39:59.209811 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1120,"mutex_wait_us":148,"thread_start_us":85,"threads_started":1}
I20260924 10:39:59.213066 6569 tablet_service.cc:1467] Processing CreateTablet for tablet d104929cc0424fb395e0dc2838ea525e (DEFAULT_TABLE table=test-table [id=37bb824521c544d8a2f190762d4bff40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:59.213157 6436 tablet_service.cc:1467] Processing CreateTablet for tablet d104929cc0424fb395e0dc2838ea525e (DEFAULT_TABLE table=test-table [id=37bb824521c544d8a2f190762d4bff40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:59.213621 6303 tablet_service.cc:1467] Processing CreateTablet for tablet d104929cc0424fb395e0dc2838ea525e (DEFAULT_TABLE table=test-table [id=37bb824521c544d8a2f190762d4bff40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:39:59.214155 6436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d104929cc0424fb395e0dc2838ea525e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:59.214165 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d104929cc0424fb395e0dc2838ea525e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:59.214497 6303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d104929cc0424fb395e0dc2838ea525e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:59.220460 6652 tablet_bootstrap.cc:492] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8: Bootstrap starting.
I20260924 10:39:59.221701 6653 tablet_bootstrap.cc:492] T d104929cc0424fb395e0dc2838ea525e P 16167dd7162549ee9a03278d53fbbb3b: Bootstrap starting.
I20260924 10:39:59.222924 6652 tablet_bootstrap.cc:654] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:59.223460 6653 tablet_bootstrap.cc:654] T d104929cc0424fb395e0dc2838ea525e P 16167dd7162549ee9a03278d53fbbb3b: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:59.224100 6652 log.cc:824] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:59.224414 6653 log.cc:824] T d104929cc0424fb395e0dc2838ea525e P 16167dd7162549ee9a03278d53fbbb3b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:59.225082 6654 tablet_bootstrap.cc:492] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de: Bootstrap starting.
I20260924 10:39:59.226238 6652 tablet_bootstrap.cc:492] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8: No bootstrap required, opened a new log
I20260924 10:39:59.226306 6653 tablet_bootstrap.cc:492] T d104929cc0424fb395e0dc2838ea525e P 16167dd7162549ee9a03278d53fbbb3b: No bootstrap required, opened a new log
I20260924 10:39:59.226425 6652 ts_tablet_manager.cc:1397] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:39:59.226563 6653 ts_tablet_manager.cc:1397] T d104929cc0424fb395e0dc2838ea525e P 16167dd7162549ee9a03278d53fbbb3b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:39:59.227160 6654 tablet_bootstrap.cc:654] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:59.228356 6654 log.cc:824] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:59.230407 6654 tablet_bootstrap.cc:492] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de: No bootstrap required, opened a new log
I20260924 10:39:59.230418 6653 raft_consensus.cc:348] T d104929cc0424fb395e0dc2838ea525e P 16167dd7162549ee9a03278d53fbbb3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297c88fc608471d9ded8df50d3fc3e8" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41457 } } peers { permanent_uuid: "67fc7010bb754a44abacbc6ff9f911de" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43599 } } peers { permanent_uuid: "16167dd7162549ee9a03278d53fbbb3b" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 37479 } }
I20260924 10:39:59.230629 6654 ts_tablet_manager.cc:1397] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260924 10:39:59.230424 6652 raft_consensus.cc:348] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297c88fc608471d9ded8df50d3fc3e8" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41457 } } peers { permanent_uuid: "67fc7010bb754a44abacbc6ff9f911de" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43599 } } peers { permanent_uuid: "16167dd7162549ee9a03278d53fbbb3b" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 37479 } }
I20260924 10:39:59.230706 6652 raft_consensus.cc:374] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:59.230710 6653 raft_consensus.cc:374] T d104929cc0424fb395e0dc2838ea525e P 16167dd7162549ee9a03278d53fbbb3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:59.230762 6653 raft_consensus.cc:729] T d104929cc0424fb395e0dc2838ea525e P 16167dd7162549ee9a03278d53fbbb3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16167dd7162549ee9a03278d53fbbb3b, State: Initialized, Role: FOLLOWER
I20260924 10:39:59.230789 6652 raft_consensus.cc:729] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297c88fc608471d9ded8df50d3fc3e8, State: Initialized, Role: FOLLOWER
I20260924 10:39:59.231266 6652 consensus_queue.cc:260] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297c88fc608471d9ded8df50d3fc3e8" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41457 } } peers { permanent_uuid: "67fc7010bb754a44abacbc6ff9f911de" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43599 } } peers { permanent_uuid: "16167dd7162549ee9a03278d53fbbb3b" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 37479 } }
I20260924 10:39:59.231280 6653 consensus_queue.cc:260] T d104929cc0424fb395e0dc2838ea525e P 16167dd7162549ee9a03278d53fbbb3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297c88fc608471d9ded8df50d3fc3e8" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41457 } } peers { permanent_uuid: "67fc7010bb754a44abacbc6ff9f911de" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43599 } } peers { permanent_uuid: "16167dd7162549ee9a03278d53fbbb3b" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 37479 } }
I20260924 10:39:59.231910 6502 heartbeater.cc:503] Master 127.2.8.62:35893 was elected leader, sending a full tablet report...
I20260924 10:39:59.232259 6653 ts_tablet_manager.cc:1428] T d104929cc0424fb395e0dc2838ea525e P 16167dd7162549ee9a03278d53fbbb3b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:39:59.232750 6369 heartbeater.cc:503] Master 127.2.8.62:35893 was elected leader, sending a full tablet report...
I20260924 10:39:59.233403 6652 ts_tablet_manager.cc:1428] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260924 10:39:59.234280 6654 raft_consensus.cc:348] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297c88fc608471d9ded8df50d3fc3e8" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41457 } } peers { permanent_uuid: "67fc7010bb754a44abacbc6ff9f911de" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43599 } } peers { permanent_uuid: "16167dd7162549ee9a03278d53fbbb3b" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 37479 } }
I20260924 10:39:59.234637 6654 raft_consensus.cc:374] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:59.234727 6654 raft_consensus.cc:729] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67fc7010bb754a44abacbc6ff9f911de, State: Initialized, Role: FOLLOWER
I20260924 10:39:59.235183 6654 consensus_queue.cc:260] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297c88fc608471d9ded8df50d3fc3e8" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41457 } } peers { permanent_uuid: "67fc7010bb754a44abacbc6ff9f911de" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43599 } } peers { permanent_uuid: "16167dd7162549ee9a03278d53fbbb3b" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 37479 } }
I20260924 10:39:59.235806 6635 heartbeater.cc:503] Master 127.2.8.62:35893 was elected leader, sending a full tablet report...
I20260924 10:39:59.236124 6654 ts_tablet_manager.cc:1428] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:39:59.256500 6658 raft_consensus.cc:482] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:39:59.256726 6658 raft_consensus.cc:504] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297c88fc608471d9ded8df50d3fc3e8" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41457 } } peers { permanent_uuid: "67fc7010bb754a44abacbc6ff9f911de" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43599 } } peers { permanent_uuid: "16167dd7162549ee9a03278d53fbbb3b" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 37479 } }
I20260924 10:39:59.257980 6658 leader_election.cc:288] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67fc7010bb754a44abacbc6ff9f911de (127.2.8.3:43599), 16167dd7162549ee9a03278d53fbbb3b (127.2.8.1:37479)
I20260924 10:39:59.262867 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.258306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:47137 (local address 127.2.8.3:43599)
0924 10:39:59.258436 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:59.258441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:59.258562 (+ 121us) server_negotiation.cc:407] Connection header received
0924 10:39:59.258631 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:59.258634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:59.258679 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:59.258791 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:59.259085 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.259935 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.261555 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.262244 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.262351 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:59.262428 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:59.262653 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 10:39:59.262729 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:39:59.262867 6651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.258352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:38357 (local address 127.2.8.1:37479)
0924 10:39:59.258495 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:59.258500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:59.258744 (+ 244us) server_negotiation.cc:407] Connection header received
0924 10:39:59.258832 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:59.258835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:59.258902 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:59.258976 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:39:59.259300 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.260129 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.261555 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.262228 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.262397 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:59.262458 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:59.262652 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 10:39:59.262736 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:39:59.262880 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.258215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:43599 (local address 127.2.8.2:47137)
0924 10:39:59.258415 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:59.258440 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:59.258535 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:59.258816 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:59.258822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:59.258836 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:59.258941 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:59.258945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.260048 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.260051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:59.261411 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:59.261422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.262368 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.262373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:59.262495 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:59.262548 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 10:39:59.262619 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 10:39:59.262720 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":65}
I20260924 10:39:59.262888 6661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.258260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:37479 (local address 127.2.8.2:38357)
0924 10:39:59.258631 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:59.258643 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:59.258728 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:59.259007 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:59.259010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:59.259076 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:39:59.259171 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:59.259176 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.260229 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.260232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:59.261411 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:39:59.261422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.262389 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.262392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:39:59.262492 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:59.262539 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 10:39:59.262619 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 10:39:59.262724 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":36,"thread_start_us":158,"threads_started":1}
I20260924 10:39:59.263624 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d104929cc0424fb395e0dc2838ea525e" candidate_uuid: "5297c88fc608471d9ded8df50d3fc3e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67fc7010bb754a44abacbc6ff9f911de" is_pre_election: true
I20260924 10:39:59.263626 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d104929cc0424fb395e0dc2838ea525e" candidate_uuid: "5297c88fc608471d9ded8df50d3fc3e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16167dd7162549ee9a03278d53fbbb3b" is_pre_election: true
I20260924 10:39:59.263949 6323 raft_consensus.cc:2454] T d104929cc0424fb395e0dc2838ea525e P 16167dd7162549ee9a03278d53fbbb3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297c88fc608471d9ded8df50d3fc3e8 in term 0.
I20260924 10:39:59.263949 6589 raft_consensus.cc:2454] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297c88fc608471d9ded8df50d3fc3e8 in term 0.
I20260924 10:39:59.264509 6389 leader_election.cc:302] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16167dd7162549ee9a03278d53fbbb3b, 5297c88fc608471d9ded8df50d3fc3e8, 67fc7010bb754a44abacbc6ff9f911de; no voters:
I20260924 10:39:59.264834 6658 raft_consensus.cc:2781] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:39:59.264942 6658 raft_consensus.cc:482] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:39:59.265039 6658 raft_consensus.cc:3037] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:59.266055 6658 raft_consensus.cc:504] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297c88fc608471d9ded8df50d3fc3e8" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41457 } } peers { permanent_uuid: "67fc7010bb754a44abacbc6ff9f911de" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43599 } } peers { permanent_uuid: "16167dd7162549ee9a03278d53fbbb3b" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 37479 } }
I20260924 10:39:59.266451 6658 leader_election.cc:288] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [CANDIDATE]: Term 1 election: Requested vote from peers 67fc7010bb754a44abacbc6ff9f911de (127.2.8.3:43599), 16167dd7162549ee9a03278d53fbbb3b (127.2.8.1:37479)
I20260924 10:39:59.266762 6589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d104929cc0424fb395e0dc2838ea525e" candidate_uuid: "5297c88fc608471d9ded8df50d3fc3e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67fc7010bb754a44abacbc6ff9f911de"
I20260924 10:39:59.266793 6323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d104929cc0424fb395e0dc2838ea525e" candidate_uuid: "5297c88fc608471d9ded8df50d3fc3e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16167dd7162549ee9a03278d53fbbb3b"
I20260924 10:39:59.266892 6589 raft_consensus.cc:3037] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:59.266909 6323 raft_consensus.cc:3037] T d104929cc0424fb395e0dc2838ea525e P 16167dd7162549ee9a03278d53fbbb3b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:59.267987 6589 raft_consensus.cc:2454] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297c88fc608471d9ded8df50d3fc3e8 in term 1.
I20260924 10:39:59.268100 6323 raft_consensus.cc:2454] T d104929cc0424fb395e0dc2838ea525e P 16167dd7162549ee9a03278d53fbbb3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297c88fc608471d9ded8df50d3fc3e8 in term 1.
I20260924 10:39:59.268365 6392 leader_election.cc:302] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297c88fc608471d9ded8df50d3fc3e8, 67fc7010bb754a44abacbc6ff9f911de; no voters:
I20260924 10:39:59.268559 6658 raft_consensus.cc:2781] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:59.268787 6658 raft_consensus.cc:686] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [term 1 LEADER]: Becoming Leader. State: Replica: 5297c88fc608471d9ded8df50d3fc3e8, State: Running, Role: LEADER
I20260924 10:39:59.269186 6658 consensus_queue.cc:237] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297c88fc608471d9ded8df50d3fc3e8" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41457 } } peers { permanent_uuid: "67fc7010bb754a44abacbc6ff9f911de" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43599 } } peers { permanent_uuid: "16167dd7162549ee9a03278d53fbbb3b" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 37479 } }
I20260924 10:39:59.272154 6179 catalog_manager.cc:5495] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297c88fc608471d9ded8df50d3fc3e8 (127.2.8.2). New cstate: current_term: 1 leader_uuid: "5297c88fc608471d9ded8df50d3fc3e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297c88fc608471d9ded8df50d3fc3e8" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67fc7010bb754a44abacbc6ff9f911de" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16167dd7162549ee9a03278d53fbbb3b" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 37479 } health_report { overall_health: UNKNOWN } } }
I20260924 10:39:59.291031 6648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.287310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.2.8.2:41457)
0924 10:39:59.287462 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:59.287467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:59.287481 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:39:59.287613 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:59.287616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:59.287680 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:59.287792 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:39:59.288057 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.288563 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.289528 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:39:59.289758 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:39:59.289867 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:39:59.290215 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:39:59.290639 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:39:59.290776 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:59.290815 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 10:39:59.290869 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:39:59.296924 6589 raft_consensus.cc:1261] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de [term 1 FOLLOWER]: Refusing update from remote peer 5297c88fc608471d9ded8df50d3fc3e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:59.297024 6323 raft_consensus.cc:1261] T d104929cc0424fb395e0dc2838ea525e P 16167dd7162549ee9a03278d53fbbb3b [term 1 FOLLOWER]: Refusing update from remote peer 5297c88fc608471d9ded8df50d3fc3e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:39:59.297636 6658 consensus_queue.cc:1035] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67fc7010bb754a44abacbc6ff9f911de" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:59.297860 6662 consensus_queue.cc:1035] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16167dd7162549ee9a03278d53fbbb3b" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 37479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:39:59.304535 6665 mvcc.cc:205] Tried to move back new op lower bound from 7332849251512860672 to 7332849251409145856. Current Snapshot: MvccSnapshot[applied={T|T < 7332849251512860672}]
I20260924 10:39:59.305387 6667 mvcc.cc:205] Tried to move back new op lower bound from 7332849251512860672 to 7332849251409145856. Current Snapshot: MvccSnapshot[applied={T|T < 7332849251512860672}]
I20260924 10:39:59.308264 6666 mvcc.cc:205] Tried to move back new op lower bound from 7332849251512860672 to 7332849251409145856. Current Snapshot: MvccSnapshot[applied={T|T < 7332849251512860672 or (T in {7332849251512860672})}]
I20260924 10:39:59.316214 6177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:39:59.316395 6177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:39:59.318711 6177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71df6df6f05041d2a71df7255e29a035: Sending DeleteTablet for 3 replicas of tablet d104929cc0424fb395e0dc2838ea525e
I20260924 10:39:59.319422 6436 tablet_service.cc:1514] Processing DeleteTablet for tablet d104929cc0424fb395e0dc2838ea525e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:59 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:44970
I20260924 10:39:59.319664 6569 tablet_service.cc:1514] Processing DeleteTablet for tablet d104929cc0424fb395e0dc2838ea525e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:59 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:37456
I20260924 10:39:59.319803 6303 tablet_service.cc:1514] Processing DeleteTablet for tablet d104929cc0424fb395e0dc2838ea525e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:39:59 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:45908
I20260924 10:39:59.320294 6675 tablet_replica.cc:330] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8: stopping tablet replica
I20260924 10:39:59.320312 6676 tablet_replica.cc:330] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de: stopping tablet replica
I20260924 10:39:59.320622 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 6241
I20260924 10:39:59.320636 6675 raft_consensus.cc:2229] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:39:59.320637 6676 raft_consensus.cc:2229] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:39:59.320933 6676 raft_consensus.cc:2258] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:59.321527 6675 raft_consensus.cc:2258] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:39:59.322806 6676 ts_tablet_manager.cc:1905] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:59.322853 6675 ts_tablet_manager.cc:1905] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:39:59.325526 6675 ts_tablet_manager.cc:1918] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:39:59.325685 6675 log.cc:1196] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/d104929cc0424fb395e0dc2838ea525e
I20260924 10:39:59.325886 6676 ts_tablet_manager.cc:1918] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:39:59.326057 6676 log.cc:1196] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/d104929cc0424fb395e0dc2838ea525e
I20260924 10:39:59.326071 6675 ts_tablet_manager.cc:1939] T d104929cc0424fb395e0dc2838ea525e P 5297c88fc608471d9ded8df50d3fc3e8: Deleting consensus metadata
I20260924 10:39:59.326470 6676 ts_tablet_manager.cc:1939] T d104929cc0424fb395e0dc2838ea525e P 67fc7010bb754a44abacbc6ff9f911de: Deleting consensus metadata
I20260924 10:39:59.327191 6165 catalog_manager.cc:4841] TS 5297c88fc608471d9ded8df50d3fc3e8 (127.2.8.2:41457): tablet d104929cc0424fb395e0dc2838ea525e (table test-table [id=37bb824521c544d8a2f190762d4bff40]) successfully deleted
I20260924 10:39:59.327733 6166 catalog_manager.cc:4841] TS 67fc7010bb754a44abacbc6ff9f911de (127.2.8.3:43599): tablet d104929cc0424fb395e0dc2838ea525e (table test-table [id=37bb824521c544d8a2f190762d4bff40]) successfully deleted
W20260924 10:39:59.329442 6163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.8.1:37479 (error 108)
I20260924 10:39:59.329555 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 6374
I20260924 10:39:59.331580 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.331310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:37479 (local address 127.0.0.1:45914)
0924 10:39:59.331451 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:59.331510 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:37479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 10:39:59.331732 6163 catalog_manager.cc:4584] TS 16167dd7162549ee9a03278d53fbbb3b (127.2.8.1:37479): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d104929cc0424fb395e0dc2838ea525e: Network error: Client connection negotiation failed: client connection to 127.2.8.1:37479: connect: Connection refused (error 111)
I20260924 10:39:59.335624 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 6507
I20260924 10:39:59.340760 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 6148
2026-09-24T10:39:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2300 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T10:39:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:39:59Z Disabled control of system clock
I20260924 10:39:59.370225 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:41615
--webserver_interface=127.2.8.62
--webserver_port=0
--builtin_ntp_servers=127.2.8.20:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:41615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 10:39:59.477186 6682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:59.477439 6682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:59.477475 6682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:59.481544 6682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:39:59.481602 6682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:59.481617 6682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:59.481634 6682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:39:59.481691 6682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:39:59.486460 6682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:41615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:41615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:59.487517 6682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:59.488891 6682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:59.497251 6690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:59.497267 6687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:59.497494 6682 server_base.cc:1034] running on GCE node
W20260924 10:39:59.497269 6688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:59.498137 6682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:59.499243 6682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:59.500437 6682 hybrid_clock.cc:648] HybridClock initialized: now 1790246399500419 us; error 30 us; skew 500 ppm
I20260924 10:39:59.502454 6682 webserver.cc:460] Webserver started at http://127.2.8.62:45793/ using document root <none> and password file <none>
I20260924 10:39:59.503051 6682 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:59.503145 6682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:59.503361 6682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:59.505514 6682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "631a303fa32f456c8b931daf389a7c3e"
format_stamp: "Formatted at 2026-09-24 10:39:59 on dist-test-slave-j38h"
server_key: "a61ff2fd908e9151a94ec00a02373c75"
server_key_iv: "ce68107e4defca048c8c68555308ec37"
server_key_version: "encryptionkey@0"
I20260924 10:39:59.506074 6682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "631a303fa32f456c8b931daf389a7c3e"
format_stamp: "Formatted at 2026-09-24 10:39:59 on dist-test-slave-j38h"
server_key: "a61ff2fd908e9151a94ec00a02373c75"
server_key_iv: "ce68107e4defca048c8c68555308ec37"
server_key_version: "encryptionkey@0"
I20260924 10:39:59.510183 6682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:39:59.512526 6696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:59.513823 6682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:39:59.513978 6682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "631a303fa32f456c8b931daf389a7c3e"
format_stamp: "Formatted at 2026-09-24 10:39:59 on dist-test-slave-j38h"
server_key: "a61ff2fd908e9151a94ec00a02373c75"
server_key_iv: "ce68107e4defca048c8c68555308ec37"
server_key_version: "encryptionkey@0"
I20260924 10:39:59.514091 6682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:59.518687 6682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:59.519310 6682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:59.519487 6682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:59.526430 6682 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:41615
I20260924 10:39:59.526441 6748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:41615 every 8 connection(s)
I20260924 10:39:59.527346 6682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:39:59.530498 6749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:39:59.535652 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 6682
I20260924 10:39:59.535787 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:39:59.536002 6749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e: Bootstrap starting.
I20260924 10:39:59.536082 2080 external_mini_cluster.cc:1428] Setting key 8c35d8d7baa4bb7b8364ea20281d165f
I20260924 10:39:59.538509 6749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e: Neither blocks nor log segments found. Creating new log.
I20260924 10:39:59.539247 6749 log.cc:824] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:39:59.541471 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.537414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.2.8.62:41615)
0924 10:39:59.538022 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:59.538037 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:59.538073 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:39:59.538765 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:59.538784 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:59.539142 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:59.539480 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:59.540186 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:59.540190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:59.540202 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:59.540210 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:59.540261 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:59.540276 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:59.540393 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:59.540497 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 10:39:59.540737 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":128,"threads_started":1}
I20260924 10:39:59.541551 6749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e: No bootstrap required, opened a new log
I20260924 10:39:59.545687 6749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a303fa32f456c8b931daf389a7c3e" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41615 } }
I20260924 10:39:59.545943 6749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:39:59.546032 6749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 631a303fa32f456c8b931daf389a7c3e, State: Initialized, Role: FOLLOWER
I20260924 10:39:59.546460 6749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a303fa32f456c8b931daf389a7c3e" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41615 } }
I20260924 10:39:59.546615 6749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:39:59.546703 6749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:39:59.546797 6749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:39:59.547747 6749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a303fa32f456c8b931daf389a7c3e" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41615 } }
I20260924 10:39:59.548098 6749 leader_election.cc:302] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 631a303fa32f456c8b931daf389a7c3e; no voters:
I20260924 10:39:59.548408 6749 leader_election.cc:288] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:39:59.548604 6754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:39:59.548940 6754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [term 1 LEADER]: Becoming Leader. State: Replica: 631a303fa32f456c8b931daf389a7c3e, State: Running, Role: LEADER
I20260924 10:39:59.549353 6754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a303fa32f456c8b931daf389a7c3e" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41615 } }
I20260924 10:39:59.549750 6749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:39:59.550994 6755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "631a303fa32f456c8b931daf389a7c3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a303fa32f456c8b931daf389a7c3e" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41615 } } }
I20260924 10:39:59.551131 6755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [sys.catalog]: This master's current role is: LEADER
I20260924 10:39:59.551496 6756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 631a303fa32f456c8b931daf389a7c3e. Latest consensus state: current_term: 1 leader_uuid: "631a303fa32f456c8b931daf389a7c3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a303fa32f456c8b931daf389a7c3e" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41615 } } }
I20260924 10:39:59.551608 6756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e [sys.catalog]: This master's current role is: LEADER
W20260924 10:39:59.554191 6769 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:39:59.554257 6769 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:39:59.554337 6765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:39:59.555007 6765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:39:59.560158 6765 catalog_manager.cc:1347] Generated new cluster ID: 8194653612ad484ba649356908fe1726
I20260924 10:39:59.560225 6765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:39:59.568382 6765 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:39:59.568692 6765 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:39:59.574154 6765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e: Generated new TSK 0
I20260924 10:39:59.574852 6765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:39:59.587416 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:41615
--builtin_ntp_servers=127.2.8.20:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 10:39:59.698849 6773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:59.699096 6773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:59.699136 6773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:59.703325 6773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:59.703372 6773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:59.703460 6773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:39:59.708305 6773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:41615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:59.709393 6773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:59.710745 6773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:59.720268 6778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:59.720295 6779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:59.720264 6781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:59.720736 6773 server_base.cc:1034] running on GCE node
I20260924 10:39:59.721189 6773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:59.721843 6773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:59.723007 6773 hybrid_clock.cc:648] HybridClock initialized: now 1790246399722973 us; error 47 us; skew 500 ppm
I20260924 10:39:59.724710 6773 webserver.cc:460] Webserver started at http://127.2.8.1:37405/ using document root <none> and password file <none>
I20260924 10:39:59.725347 6773 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:59.725447 6773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:59.725661 6773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:59.727664 6773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "6c89093a84e448f78ca629a382088653"
format_stamp: "Formatted at 2026-09-24 10:39:59 on dist-test-slave-j38h"
server_key: "b5481fc0ff6b8abfef6645f8bbdb9261"
server_key_iv: "bd2b9dc5b57bb5e3510a6fb370121e3c"
server_key_version: "encryptionkey@0"
I20260924 10:39:59.728230 6773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "6c89093a84e448f78ca629a382088653"
format_stamp: "Formatted at 2026-09-24 10:39:59 on dist-test-slave-j38h"
server_key: "b5481fc0ff6b8abfef6645f8bbdb9261"
server_key_iv: "bd2b9dc5b57bb5e3510a6fb370121e3c"
server_key_version: "encryptionkey@0"
I20260924 10:39:59.731771 6773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:39:59.734102 6787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:59.735270 6773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:39:59.735401 6773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "6c89093a84e448f78ca629a382088653"
format_stamp: "Formatted at 2026-09-24 10:39:59 on dist-test-slave-j38h"
server_key: "b5481fc0ff6b8abfef6645f8bbdb9261"
server_key_iv: "bd2b9dc5b57bb5e3510a6fb370121e3c"
server_key_version: "encryptionkey@0"
I20260924 10:39:59.735529 6773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:59.740298 6773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:59.740931 6773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:59.741129 6773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:59.741659 6773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:59.742523 6773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:59.742594 6773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:59.742660 6773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:59.742714 6773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:59.751698 6773 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:44479
I20260924 10:39:59.751718 6900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:44479 every 8 connection(s)
I20260924 10:39:59.752738 6773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:39:59.753690 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 6773
I20260924 10:39:59.753813 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:39:59.754096 2080 external_mini_cluster.cc:1428] Setting key 9f6235ead541a095c54c6fd291f1b84b
I20260924 10:39:59.756224 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:41615
--builtin_ntp_servers=127.2.8.20:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 10:39:59.760143 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.756217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:34667 (local address 127.2.8.62:41615)
0924 10:39:59.756342 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:59.756345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:59.756358 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:39:59.758268 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:59.758274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:59.758323 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:59.758419 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:59.759553 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:59.759559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:59.759565 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:59.759582 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:59.759609 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:59.759622 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:59.759722 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:59.759883 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 10:39:59.759932 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:39:59.760934 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.755240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41615 (local address 127.2.8.1:34667)
0924 10:39:59.756193 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:59.756250 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:59.758062 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:59.758576 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:59.758583 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:59.758673 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:59.759288 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:39:59.759289 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:59.759331 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:39:59.759400 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:59.759711 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:59.759716 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:59.759720 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:39:59.759915 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 10:39:59.760119 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":174,"threads_started":1}
I20260924 10:39:59.762217 6901 heartbeater.cc:347] Connected to a master server at 127.2.8.62:41615
I20260924 10:39:59.762445 6901 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:59.762907 6901 heartbeater.cc:511] Master 127.2.8.62:41615 requested a full tablet report, sending...
I20260924 10:39:59.764420 6713 ts_manager.cc:194] Registered new tserver with Master: 6c89093a84e448f78ca629a382088653 (127.2.8.1:44479)
W20260924 10:39:59.867197 6904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:39:59.867487 6904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:39:59.867571 6904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:39:59.871613 6904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:39:59.871699 6904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:39:59.871790 6904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:39:59.876391 6904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:41615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:39:59.877506 6904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:39:59.878871 6904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:39:59.888049 6912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:59.888072 6909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:39:59.888053 6910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:39:59.888358 6904 server_base.cc:1034] running on GCE node
I20260924 10:39:59.888723 6904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:39:59.889394 6904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:39:59.890583 6904 hybrid_clock.cc:648] HybridClock initialized: now 1790246399890556 us; error 36 us; skew 500 ppm
I20260924 10:39:59.892520 6904 webserver.cc:460] Webserver started at http://127.2.8.2:43877/ using document root <none> and password file <none>
I20260924 10:39:59.893187 6904 fs_manager.cc:359] Metadata directory not provided
I20260924 10:39:59.893250 6904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:39:59.893486 6904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:39:59.895495 6904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "c1fbe95c5a96447f9cc80be13fb8bf81"
format_stamp: "Formatted at 2026-09-24 10:39:59 on dist-test-slave-j38h"
server_key: "f0881ed7846e7403926601f1976f699b"
server_key_iv: "4d00a391ff414a6dfdece9d6ad813d2a"
server_key_version: "encryptionkey@0"
I20260924 10:39:59.896034 6904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "c1fbe95c5a96447f9cc80be13fb8bf81"
format_stamp: "Formatted at 2026-09-24 10:39:59 on dist-test-slave-j38h"
server_key: "f0881ed7846e7403926601f1976f699b"
server_key_iv: "4d00a391ff414a6dfdece9d6ad813d2a"
server_key_version: "encryptionkey@0"
I20260924 10:39:59.899533 6904 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 10:39:59.901867 6918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:59.903108 6904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:39:59.903239 6904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "c1fbe95c5a96447f9cc80be13fb8bf81"
format_stamp: "Formatted at 2026-09-24 10:39:59 on dist-test-slave-j38h"
server_key: "f0881ed7846e7403926601f1976f699b"
server_key_iv: "4d00a391ff414a6dfdece9d6ad813d2a"
server_key_version: "encryptionkey@0"
I20260924 10:39:59.903324 6904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:39:59.907724 6904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:39:59.908232 6904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:39:59.908358 6904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:39:59.908809 6904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:39:59.909611 6904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:39:59.909654 6904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:59.909689 6904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:39:59.909713 6904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:39:59.917954 6904 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:45449
I20260924 10:39:59.917979 7031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:45449 every 8 connection(s)
I20260924 10:39:59.918830 6904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:39:59.922544 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 6904
I20260924 10:39:59.922643 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:39:59.922950 2080 external_mini_cluster.cc:1428] Setting key daa234fdae445e29b84c2bdbbd4543b1
I20260924 10:39:59.924705 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.920561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:53625 (local address 127.2.8.62:41615)
0924 10:39:59.920694 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:39:59.920698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:39:59.921743 (+ 1045us) server_negotiation.cc:407] Connection header received
0924 10:39:59.923053 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:39:59.923057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:39:59.923102 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:39:59.923183 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:39:59.924238 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:39:59.924242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:39:59.924244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:39:59.924256 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:39:59.924277 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:39:59.924286 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:39:59.924384 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:39:59.924536 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 10:39:59.924577 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:39:59.924999 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:41615
--builtin_ntp_servers=127.2.8.20:45145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 10:39:59.925673 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:39:59.920920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41615 (local address 127.2.8.2:53625)
0924 10:39:59.921606 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 10:39:59.921644 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:39:59.922844 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:39:59.923346 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:39:59.923352 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:39:59.923443 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:39:59.924001 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:39:59.924003 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:39:59.924041 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:39:59.924103 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:39:59.924380 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:39:59.924389 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:39:59.924396 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:39:59.924578 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 10:39:59.924810 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":105,"threads_started":1}
I20260924 10:39:59.926786 7032 heartbeater.cc:347] Connected to a master server at 127.2.8.62:41615
I20260924 10:39:59.927033 7032 heartbeater.cc:464] Registering TS with master...
I20260924 10:39:59.927490 7032 heartbeater.cc:511] Master 127.2.8.62:41615 requested a full tablet report, sending...
I20260924 10:39:59.928543 6713 ts_manager.cc:194] Registered new tserver with Master: c1fbe95c5a96447f9cc80be13fb8bf81 (127.2.8.2:45449)
W20260924 10:40:00.034969 7035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:00.035229 7035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:00.035269 7035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:00.039410 7035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:00.039474 7035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:00.039529 7035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:40:00.044402 7035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:41615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:00.045552 7035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:00.046872 7035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:00.056576 7043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:00.056520 7040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:00.056674 7035 server_base.cc:1034] running on GCE node
W20260924 10:40:00.056869 7041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:00.057271 7035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:00.057888 7035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:00.059043 7035 hybrid_clock.cc:648] HybridClock initialized: now 1790246400059011 us; error 44 us; skew 500 ppm
I20260924 10:40:00.061064 7035 webserver.cc:460] Webserver started at http://127.2.8.3:40175/ using document root <none> and password file <none>
I20260924 10:40:00.061645 7035 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:00.061736 7035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:00.061970 7035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:00.064076 7035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "9d0b77c465834f8ea4b8af454276a876"
format_stamp: "Formatted at 2026-09-24 10:40:00 on dist-test-slave-j38h"
server_key: "de796df62bd61044f114c76c5b651993"
server_key_iv: "0c32759495c25bf466bd7f85268f9870"
server_key_version: "encryptionkey@0"
I20260924 10:40:00.064648 7035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "9d0b77c465834f8ea4b8af454276a876"
format_stamp: "Formatted at 2026-09-24 10:40:00 on dist-test-slave-j38h"
server_key: "de796df62bd61044f114c76c5b651993"
server_key_iv: "0c32759495c25bf466bd7f85268f9870"
server_key_version: "encryptionkey@0"
I20260924 10:40:00.068239 7035 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:40:00.070660 7049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:00.071884 7035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:40:00.072005 7035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "9d0b77c465834f8ea4b8af454276a876"
format_stamp: "Formatted at 2026-09-24 10:40:00 on dist-test-slave-j38h"
server_key: "de796df62bd61044f114c76c5b651993"
server_key_iv: "0c32759495c25bf466bd7f85268f9870"
server_key_version: "encryptionkey@0"
I20260924 10:40:00.072109 7035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:00.076730 7035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:00.077374 7035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:00.077559 7035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:00.078099 7035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:00.078989 7035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:00.079032 7035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:00.079098 7035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:00.079135 7035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:00.087764 7035 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:43309
I20260924 10:40:00.087782 7162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:43309 every 8 connection(s)
I20260924 10:40:00.088631 7035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:40:00.091166 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 7035
I20260924 10:40:00.091303 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:40:00.091567 2080 external_mini_cluster.cc:1428] Setting key f45347dc01fc3a6edb3eed46714f33b9
I20260924 10:40:00.094748 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.090540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:45377 (local address 127.2.8.62:41615)
0924 10:40:00.090678 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:00.090681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:00.091757 (+ 1076us) server_negotiation.cc:407] Connection header received
0924 10:40:00.093167 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:00.093170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:00.093204 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:00.093261 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:00.094407 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:00.094409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:00.094411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:00.094422 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:00.094441 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:00.094449 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:00.094513 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:00.094614 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 10:40:00.094649 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:40:00.095566 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.090854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41615 (local address 127.2.8.3:45377)
0924 10:40:00.091590 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:00.091640 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:00.092909 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:00.093426 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:00.093447 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:00.093569 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:00.094166 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:40:00.094167 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:00.094206 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:40:00.094270 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:00.094510 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:00.094514 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:00.094518 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:40:00.094631 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 10:40:00.094810 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":151,"threads_started":1}
I20260924 10:40:00.096738 7163 heartbeater.cc:347] Connected to a master server at 127.2.8.62:41615
I20260924 10:40:00.097025 7163 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:00.097407 7163 heartbeater.cc:511] Master 127.2.8.62:41615 requested a full tablet report, sending...
I20260924 10:40:00.098325 6713 ts_manager.cc:194] Registered new tserver with Master: 9d0b77c465834f8ea4b8af454276a876 (127.2.8.3:43309)
I20260924 10:40:00.105396 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:40:00.109139 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.107425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.2.8.62:41615)
0924 10:40:00.107551 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:00.107555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:00.107749 (+ 194us) server_negotiation.cc:407] Connection header received
0924 10:40:00.107940 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:00.107943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:00.107974 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:00.108038 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:00.108780 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:00.108782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:00.108784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:00.108792 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:00.108807 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:00.108814 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:00.108876 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:00.108956 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 10:40:00.109029 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:40:00.111680 6713 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:40:00.114607 6713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:40:00.125744 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.122720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:45449 (local address 127.0.0.1:54902)
0924 10:40:00.123348 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:00.123401 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:00.123508 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:00.124481 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:00.124483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:00.124497 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:00.124933 (+ 436us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:40:00.124935 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:00.124943 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:40:00.124962 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:00.125371 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:00.125383 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:00.125391 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:40:00.125510 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 10:40:00.125580 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"spinlock_wait_cycles":10368,"thread_start_us":104,"threads_started":1}
I20260924 10:40:00.125819 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.123011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54902 (local address 127.2.8.2:45449)
0924 10:40:00.124092 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:00.124104 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:00.124126 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:40:00.124177 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:00.124182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:00.124342 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:00.124495 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:00.125151 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:00.125156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:00.125163 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:00.125174 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:00.125202 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:00.125221 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:00.125324 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:00.125575 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 10:40:00.125645 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":985,"thread_start_us":128,"threads_started":1}
I20260924 10:40:00.126055 7175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.122579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.2.8.1:44479)
0924 10:40:00.123483 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:00.123496 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:00.123522 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:40:00.123588 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:00.123594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:00.123760 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:00.123916 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:00.125108 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:00.125115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:00.125130 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:00.125141 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:00.125183 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:00.125217 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:00.125324 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:00.125812 (+ 488us) server_negotiation.cc:299] Negotiation successful
0924 10:40:00.125895 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":132,"threads_started":1}
I20260924 10:40:00.125919 7174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.122411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:44479 (local address 127.0.0.1:57258)
0924 10:40:00.123335 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:00.123383 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:00.123506 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:00.123907 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:00.123913 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:00.123985 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:00.124713 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:40:00.124717 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:00.124736 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:40:00.124805 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:00.125609 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:00.125612 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:00.125615 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 10:40:00.125685 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 10:40:00.125713 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"spinlock_wait_cycles":768,"thread_start_us":83,"threads_started":1}
I20260924 10:40:00.126603 7176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.123120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:43309 (local address 127.0.0.1:49744)
0924 10:40:00.124656 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:00.124670 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:00.124743 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:00.125214 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:00.125217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:00.125230 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:00.125972 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:40:00.125974 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:00.125984 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:40:00.126012 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:00.126371 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:00.126376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:00.126380 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:40:00.126456 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 10:40:00.126491 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1462,"thread_start_us":99,"threads_started":1}
I20260924 10:40:00.127039 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.123368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49744 (local address 127.2.8.3:43309)
0924 10:40:00.123902 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:00.123913 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:00.124825 (+ 912us) server_negotiation.cc:407] Connection header received
0924 10:40:00.124894 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:00.124901 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:00.125097 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:00.125231 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:00.126127 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:00.126133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:00.126144 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:00.126155 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:00.126199 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:00.126236 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:00.126330 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:00.126819 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 10:40:00.126880 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":93,"threads_started":1}
I20260924 10:40:00.129220 6966 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6ad1b160654ea4b888b4eaa26123e2 (DEFAULT_TABLE table=test-table [id=d0dab21560644056a7329a6db7787865]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:00.129220 6835 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6ad1b160654ea4b888b4eaa26123e2 (DEFAULT_TABLE table=test-table [id=d0dab21560644056a7329a6db7787865]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:00.129451 7097 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6ad1b160654ea4b888b4eaa26123e2 (DEFAULT_TABLE table=test-table [id=d0dab21560644056a7329a6db7787865]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:00.130308 6835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6ad1b160654ea4b888b4eaa26123e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:00.130425 7097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6ad1b160654ea4b888b4eaa26123e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:00.130527 6966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6ad1b160654ea4b888b4eaa26123e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:00.136297 7180 tablet_bootstrap.cc:492] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876: Bootstrap starting.
I20260924 10:40:00.137812 7182 tablet_bootstrap.cc:492] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81: Bootstrap starting.
I20260924 10:40:00.137939 7181 tablet_bootstrap.cc:492] T 1c6ad1b160654ea4b888b4eaa26123e2 P 6c89093a84e448f78ca629a382088653: Bootstrap starting.
I20260924 10:40:00.138835 7180 tablet_bootstrap.cc:654] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:00.139711 7180 log.cc:824] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:00.140089 7181 tablet_bootstrap.cc:654] T 1c6ad1b160654ea4b888b4eaa26123e2 P 6c89093a84e448f78ca629a382088653: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:00.140089 7182 tablet_bootstrap.cc:654] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:00.140923 7182 log.cc:824] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:00.140923 7181 log.cc:824] T 1c6ad1b160654ea4b888b4eaa26123e2 P 6c89093a84e448f78ca629a382088653: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:00.141674 7180 tablet_bootstrap.cc:492] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876: No bootstrap required, opened a new log
I20260924 10:40:00.141866 7180 ts_tablet_manager.cc:1397] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260924 10:40:00.142891 7181 tablet_bootstrap.cc:492] T 1c6ad1b160654ea4b888b4eaa26123e2 P 6c89093a84e448f78ca629a382088653: No bootstrap required, opened a new log
I20260924 10:40:00.142973 7182 tablet_bootstrap.cc:492] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81: No bootstrap required, opened a new log
I20260924 10:40:00.143086 7181 ts_tablet_manager.cc:1397] T 1c6ad1b160654ea4b888b4eaa26123e2 P 6c89093a84e448f78ca629a382088653: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:40:00.143095 7182 ts_tablet_manager.cc:1397] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:40:00.145534 7180 raft_consensus.cc:348] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c89093a84e448f78ca629a382088653" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44479 } } peers { permanent_uuid: "c1fbe95c5a96447f9cc80be13fb8bf81" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45449 } } peers { permanent_uuid: "9d0b77c465834f8ea4b8af454276a876" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43309 } }
I20260924 10:40:00.145773 7180 raft_consensus.cc:374] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:00.145843 7180 raft_consensus.cc:729] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0b77c465834f8ea4b8af454276a876, State: Initialized, Role: FOLLOWER
I20260924 10:40:00.146302 7180 consensus_queue.cc:260] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c89093a84e448f78ca629a382088653" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44479 } } peers { permanent_uuid: "c1fbe95c5a96447f9cc80be13fb8bf81" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45449 } } peers { permanent_uuid: "9d0b77c465834f8ea4b8af454276a876" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43309 } }
I20260924 10:40:00.146965 7163 heartbeater.cc:503] Master 127.2.8.62:41615 was elected leader, sending a full tablet report...
I20260924 10:40:00.147379 7180 ts_tablet_manager.cc:1428] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260924 10:40:00.147637 7182 raft_consensus.cc:348] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c89093a84e448f78ca629a382088653" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44479 } } peers { permanent_uuid: "c1fbe95c5a96447f9cc80be13fb8bf81" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45449 } } peers { permanent_uuid: "9d0b77c465834f8ea4b8af454276a876" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43309 } }
I20260924 10:40:00.147586 7181 raft_consensus.cc:348] T 1c6ad1b160654ea4b888b4eaa26123e2 P 6c89093a84e448f78ca629a382088653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c89093a84e448f78ca629a382088653" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44479 } } peers { permanent_uuid: "c1fbe95c5a96447f9cc80be13fb8bf81" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45449 } } peers { permanent_uuid: "9d0b77c465834f8ea4b8af454276a876" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43309 } }
I20260924 10:40:00.147966 7181 raft_consensus.cc:374] T 1c6ad1b160654ea4b888b4eaa26123e2 P 6c89093a84e448f78ca629a382088653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:00.147966 7182 raft_consensus.cc:374] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:00.148029 7181 raft_consensus.cc:729] T 1c6ad1b160654ea4b888b4eaa26123e2 P 6c89093a84e448f78ca629a382088653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c89093a84e448f78ca629a382088653, State: Initialized, Role: FOLLOWER
I20260924 10:40:00.148029 7182 raft_consensus.cc:729] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1fbe95c5a96447f9cc80be13fb8bf81, State: Initialized, Role: FOLLOWER
I20260924 10:40:00.148479 7181 consensus_queue.cc:260] T 1c6ad1b160654ea4b888b4eaa26123e2 P 6c89093a84e448f78ca629a382088653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c89093a84e448f78ca629a382088653" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44479 } } peers { permanent_uuid: "c1fbe95c5a96447f9cc80be13fb8bf81" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45449 } } peers { permanent_uuid: "9d0b77c465834f8ea4b8af454276a876" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43309 } }
I20260924 10:40:00.149140 6901 heartbeater.cc:503] Master 127.2.8.62:41615 was elected leader, sending a full tablet report...
I20260924 10:40:00.149463 7181 ts_tablet_manager.cc:1428] T 1c6ad1b160654ea4b888b4eaa26123e2 P 6c89093a84e448f78ca629a382088653: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 10:40:00.149950 7182 consensus_queue.cc:260] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c89093a84e448f78ca629a382088653" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44479 } } peers { permanent_uuid: "c1fbe95c5a96447f9cc80be13fb8bf81" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45449 } } peers { permanent_uuid: "9d0b77c465834f8ea4b8af454276a876" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43309 } }
I20260924 10:40:00.151026 7182 ts_tablet_manager.cc:1428] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260924 10:40:00.151329 7032 heartbeater.cc:503] Master 127.2.8.62:41615 was elected leader, sending a full tablet report...
W20260924 10:40:00.169809 7033 tablet.cc:2369] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:40:00.253993 6902 tablet.cc:2369] T 1c6ad1b160654ea4b888b4eaa26123e2 P 6c89093a84e448f78ca629a382088653: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:40:00.268384 7188 raft_consensus.cc:482] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:40:00.268591 7188 raft_consensus.cc:504] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c89093a84e448f78ca629a382088653" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44479 } } peers { permanent_uuid: "c1fbe95c5a96447f9cc80be13fb8bf81" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45449 } } peers { permanent_uuid: "9d0b77c465834f8ea4b8af454276a876" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43309 } }
I20260924 10:40:00.269824 7188 leader_election.cc:288] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c89093a84e448f78ca629a382088653 (127.2.8.1:44479), 9d0b77c465834f8ea4b8af454276a876 (127.2.8.3:43309)
I20260924 10:40:00.271911 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.270154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:44479 (local address 127.2.8.2:48165)
0924 10:40:00.270294 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:00.270310 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:00.270402 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:00.270750 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:00.270753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:00.270768 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:00.271274 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:40:00.271277 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:00.271290 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:40:00.271326 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:00.271647 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:00.271651 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:00.271654 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 10:40:00.271746 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 10:40:00.271783 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 10:40:00.271911 7175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.270355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:48165 (local address 127.2.8.1:44479)
0924 10:40:00.270544 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:00.270547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:00.270563 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:40:00.270602 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:00.270604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:00.270633 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:00.270733 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:00.271481 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:00.271485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:00.271487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:00.271499 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:00.271519 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:00.271529 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:00.271626 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:00.271759 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 10:40:00.271798 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 10:40:00.272084 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.270277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:58939 (local address 127.2.8.3:43309)
0924 10:40:00.270404 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:00.270409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:00.270673 (+ 264us) server_negotiation.cc:407] Connection header received
0924 10:40:00.270765 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:00.270768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:00.270810 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:00.270897 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:00.271649 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:00.271652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:00.271655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:00.271669 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:00.271689 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:00.271699 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:00.271786 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:00.271930 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 10:40:00.271970 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:40:00.272113 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.270204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:43309 (local address 127.2.8.2:58939)
0924 10:40:00.270573 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:00.270583 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:00.270683 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:00.270914 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:00.270917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:00.270939 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:00.271483 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:40:00.271485 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:00.271496 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:40:00.271521 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:00.271829 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:00.271835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:00.271838 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 10:40:00.271928 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 10:40:00.271965 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":3,"thread_start_us":106,"threads_started":1}
I20260924 10:40:00.272657 6855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6ad1b160654ea4b888b4eaa26123e2" candidate_uuid: "c1fbe95c5a96447f9cc80be13fb8bf81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c89093a84e448f78ca629a382088653" is_pre_election: true
I20260924 10:40:00.272747 7117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6ad1b160654ea4b888b4eaa26123e2" candidate_uuid: "c1fbe95c5a96447f9cc80be13fb8bf81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0b77c465834f8ea4b8af454276a876" is_pre_election: true
I20260924 10:40:00.273046 6855 raft_consensus.cc:2454] T 1c6ad1b160654ea4b888b4eaa26123e2 P 6c89093a84e448f78ca629a382088653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1fbe95c5a96447f9cc80be13fb8bf81 in term 0.
I20260924 10:40:00.273082 7117 raft_consensus.cc:2454] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1fbe95c5a96447f9cc80be13fb8bf81 in term 0.
I20260924 10:40:00.273597 6919 leader_election.cc:302] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6c89093a84e448f78ca629a382088653, 9d0b77c465834f8ea4b8af454276a876, c1fbe95c5a96447f9cc80be13fb8bf81; no voters:
I20260924 10:40:00.273903 7188 raft_consensus.cc:2781] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:40:00.273988 7188 raft_consensus.cc:482] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:40:00.274031 7188 raft_consensus.cc:3037] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:00.275107 7188 raft_consensus.cc:504] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c89093a84e448f78ca629a382088653" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44479 } } peers { permanent_uuid: "c1fbe95c5a96447f9cc80be13fb8bf81" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45449 } } peers { permanent_uuid: "9d0b77c465834f8ea4b8af454276a876" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43309 } }
I20260924 10:40:00.275421 7188 leader_election.cc:288] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [CANDIDATE]: Term 1 election: Requested vote from peers 6c89093a84e448f78ca629a382088653 (127.2.8.1:44479), 9d0b77c465834f8ea4b8af454276a876 (127.2.8.3:43309)
I20260924 10:40:00.275707 6855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6ad1b160654ea4b888b4eaa26123e2" candidate_uuid: "c1fbe95c5a96447f9cc80be13fb8bf81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c89093a84e448f78ca629a382088653"
I20260924 10:40:00.275717 7117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6ad1b160654ea4b888b4eaa26123e2" candidate_uuid: "c1fbe95c5a96447f9cc80be13fb8bf81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0b77c465834f8ea4b8af454276a876"
I20260924 10:40:00.275830 7117 raft_consensus.cc:3037] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:00.275830 6855 raft_consensus.cc:3037] T 1c6ad1b160654ea4b888b4eaa26123e2 P 6c89093a84e448f78ca629a382088653 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:00.277124 6855 raft_consensus.cc:2454] T 1c6ad1b160654ea4b888b4eaa26123e2 P 6c89093a84e448f78ca629a382088653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1fbe95c5a96447f9cc80be13fb8bf81 in term 1.
I20260924 10:40:00.277124 7117 raft_consensus.cc:2454] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1fbe95c5a96447f9cc80be13fb8bf81 in term 1.
I20260924 10:40:00.277436 6919 leader_election.cc:302] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0b77c465834f8ea4b8af454276a876, c1fbe95c5a96447f9cc80be13fb8bf81; no voters:
I20260924 10:40:00.277618 7188 raft_consensus.cc:2781] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:00.277858 7188 raft_consensus.cc:686] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [term 1 LEADER]: Becoming Leader. State: Replica: c1fbe95c5a96447f9cc80be13fb8bf81, State: Running, Role: LEADER
I20260924 10:40:00.278194 7188 consensus_queue.cc:237] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c89093a84e448f78ca629a382088653" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44479 } } peers { permanent_uuid: "c1fbe95c5a96447f9cc80be13fb8bf81" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45449 } } peers { permanent_uuid: "9d0b77c465834f8ea4b8af454276a876" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43309 } }
I20260924 10:40:00.281342 6711 catalog_manager.cc:5495] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1fbe95c5a96447f9cc80be13fb8bf81 (127.2.8.2). New cstate: current_term: 1 leader_uuid: "c1fbe95c5a96447f9cc80be13fb8bf81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c89093a84e448f78ca629a382088653" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1fbe95c5a96447f9cc80be13fb8bf81" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d0b77c465834f8ea4b8af454276a876" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43309 } health_report { overall_health: UNKNOWN } } }
I20260924 10:40:00.316316 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.314858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.2.8.2:45449)
0924 10:40:00.314991 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:00.314997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:00.315016 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:40:00.315109 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:00.315111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:00.315167 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:00.315259 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:00.315883 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:00.315887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:00.315889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:00.315901 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:00.315921 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:00.315935 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:00.316024 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:00.316129 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 10:40:00.316169 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:40:00.322096 7117 raft_consensus.cc:1261] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876 [term 1 FOLLOWER]: Refusing update from remote peer c1fbe95c5a96447f9cc80be13fb8bf81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:00.322096 6855 raft_consensus.cc:1261] T 1c6ad1b160654ea4b888b4eaa26123e2 P 6c89093a84e448f78ca629a382088653 [term 1 FOLLOWER]: Refusing update from remote peer c1fbe95c5a96447f9cc80be13fb8bf81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:00.322842 7190 consensus_queue.cc:1035] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0b77c465834f8ea4b8af454276a876" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:00.323026 7188 consensus_queue.cc:1035] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c89093a84e448f78ca629a382088653" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:00.330507 7194 mvcc.cc:205] Tried to move back new op lower bound from 7332849255712215040 to 7332849255542136832. Current Snapshot: MvccSnapshot[applied={T|T < 7332849255712215040}]
I20260924 10:40:00.339308 6713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:40:00.339469 6713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:40:00.341950 6713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 631a303fa32f456c8b931daf389a7c3e: Sending DeleteTablet for 3 replicas of tablet 1c6ad1b160654ea4b888b4eaa26123e2
I20260924 10:40:00.342684 6966 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6ad1b160654ea4b888b4eaa26123e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:00 UTC) from {username='slave'} at 127.0.0.1:54902
I20260924 10:40:00.342689 7097 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6ad1b160654ea4b888b4eaa26123e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:00 UTC) from {username='slave'} at 127.0.0.1:49744
I20260924 10:40:00.342689 6835 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6ad1b160654ea4b888b4eaa26123e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:00 UTC) from {username='slave'} at 127.0.0.1:57258
I20260924 10:40:00.343144 7202 tablet_replica.cc:330] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81: stopping tablet replica
I20260924 10:40:00.343452 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 6773
I20260924 10:40:00.343488 7202 raft_consensus.cc:2229] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:40:00.344238 7202 raft_consensus.cc:2258] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:00.345651 7204 tablet_replica.cc:330] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876: stopping tablet replica
I20260924 10:40:00.345701 7202 ts_tablet_manager.cc:1905] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:00.345952 7204 raft_consensus.cc:2229] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:40:00.346251 7204 raft_consensus.cc:2258] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:00.347733 7204 ts_tablet_manager.cc:1905] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:00.348626 7202 ts_tablet_manager.cc:1918] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:40:00.348769 7202 log.cc:1196] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/1c6ad1b160654ea4b888b4eaa26123e2
I20260924 10:40:00.349215 7202 ts_tablet_manager.cc:1939] T 1c6ad1b160654ea4b888b4eaa26123e2 P c1fbe95c5a96447f9cc80be13fb8bf81: Deleting consensus metadata
I20260924 10:40:00.350520 7204 ts_tablet_manager.cc:1918] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:40:00.350584 6700 catalog_manager.cc:4841] TS c1fbe95c5a96447f9cc80be13fb8bf81 (127.2.8.2:45449): tablet 1c6ad1b160654ea4b888b4eaa26123e2 (table test-table [id=d0dab21560644056a7329a6db7787865]) successfully deleted
I20260924 10:40:00.350685 7204 log.cc:1196] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/1c6ad1b160654ea4b888b4eaa26123e2
I20260924 10:40:00.351078 7204 ts_tablet_manager.cc:1939] T 1c6ad1b160654ea4b888b4eaa26123e2 P 9d0b77c465834f8ea4b8af454276a876: Deleting consensus metadata
W20260924 10:40:00.351765 6699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.8.1:44479 (error 108)
I20260924 10:40:00.351807 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 6904
I20260924 10:40:00.353065 6697 catalog_manager.cc:4841] TS 9d0b77c465834f8ea4b8af454276a876 (127.2.8.3:43309): tablet 1c6ad1b160654ea4b888b4eaa26123e2 (table test-table [id=d0dab21560644056a7329a6db7787865]) successfully deleted
I20260924 10:40:00.354198 7176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.353835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:44479 (local address 127.0.0.1:51788)
0924 10:40:00.354007 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:00.354091 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:44479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 10:40:00.354377 6699 catalog_manager.cc:4584] TS 6c89093a84e448f78ca629a382088653 (127.2.8.1:44479): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c6ad1b160654ea4b888b4eaa26123e2: Network error: Client connection negotiation failed: client connection to 127.2.8.1:44479: connect: Connection refused (error 111)
I20260924 10:40:00.357829 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 7035
I20260924 10:40:00.363679 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 6682
2026-09-24T10:40:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1021 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 10:40:00.384908 2080 mini_oidc.cc:152] Starting JWKS server
I20260924 10:40:00.386551 2080 webserver.cc:460] Webserver started at https://127.0.0.1:41461/ using document root <none> and password file <none>
I20260924 10:40:00.386686 2080 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 10:40:00.386950 2080 webserver.cc:460] Webserver started at http://127.0.0.1:38143/ using document root <none> and password file <none>
2026-09-24T10:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:40:00Z Disabled control of system clock
I20260924 10:40:00.395519 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:45917
--webserver_interface=127.2.8.62
--webserver_port=0
--builtin_ntp_servers=127.2.8.20:40613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:45917
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41461/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/testchainca.cert with env {}
W20260924 10:40:00.509845 7211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:00.510227 7211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:00.510372 7211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:00.517289 7211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:40:00.517390 7211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:00.517421 7211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:00.517467 7211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:40:00.517493 7211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:40:00.517519 7211 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:40:00.517552 7211 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41461/jwks.json
I20260924 10:40:00.527084 7211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:40613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:45917
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:45917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41461/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:00.528267 7211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:00.529870 7211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:00.538722 7216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:00.538722 7217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:00.538880 7211 server_base.cc:1034] running on GCE node
W20260924 10:40:00.538970 7219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:00.539849 7211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:00.541390 7211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:00.542591 7211 hybrid_clock.cc:648] HybridClock initialized: now 1790246400542564 us; error 50 us; skew 500 ppm
I20260924 10:40:00.544826 7211 webserver.cc:460] Webserver started at http://127.2.8.62:45331/ using document root <none> and password file <none>
I20260924 10:40:00.545642 7211 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:00.545728 7211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:00.545933 7211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:00.548198 7211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "95d68790e44f40fdb8ddef524a6d0921"
format_stamp: "Formatted at 2026-09-24 10:40:00 on dist-test-slave-j38h"
server_key: "4f8ab41e61fe45a669b48fd254077891"
server_key_iv: "87bf56829bab63f6003be118ef032496"
server_key_version: "encryptionkey@0"
I20260924 10:40:00.548768 7211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "95d68790e44f40fdb8ddef524a6d0921"
format_stamp: "Formatted at 2026-09-24 10:40:00 on dist-test-slave-j38h"
server_key: "4f8ab41e61fe45a669b48fd254077891"
server_key_iv: "87bf56829bab63f6003be118ef032496"
server_key_version: "encryptionkey@0"
I20260924 10:40:00.552639 7211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:40:00.555160 7225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:00.556550 7211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:40:00.556690 7211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "95d68790e44f40fdb8ddef524a6d0921"
format_stamp: "Formatted at 2026-09-24 10:40:00 on dist-test-slave-j38h"
server_key: "4f8ab41e61fe45a669b48fd254077891"
server_key_iv: "87bf56829bab63f6003be118ef032496"
server_key_version: "encryptionkey@0"
I20260924 10:40:00.556806 7211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:00.586369 7211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:00.587235 7211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:00.587431 7211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:00.595629 7211 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:45917
I20260924 10:40:00.595701 7277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:45917 every 8 connection(s)
I20260924 10:40:00.596712 7211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:40:00.600179 7278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:00.601400 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 7211
I20260924 10:40:00.601528 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:40:00.601882 2080 external_mini_cluster.cc:1428] Setting key 65a09e344bd46f8c439ea5f87e2d52bb
I20260924 10:40:00.606928 7278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921: Bootstrap starting.
I20260924 10:40:00.609440 7278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:00.610399 7278 log.cc:824] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:00.611579 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.603297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37258 (local address 127.2.8.62:45917)
0924 10:40:00.603952 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:00.603966 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:00.604001 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:40:00.604636 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:00.604653 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:00.605138 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:00.605471 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:00.605963 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:00.606943 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:00.607824 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:00.608302 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:00.608475 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:00.609402 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:00.609411 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:00.609430 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:00.609989 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:00.610061 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:00.610088 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:00.610211 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:00.610333 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 10:40:00.610620 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":125,"threads_started":1}
I20260924 10:40:00.612654 7278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921: No bootstrap required, opened a new log
I20260924 10:40:00.614450 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:40:00.616955 7278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d68790e44f40fdb8ddef524a6d0921" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45917 } }
I20260924 10:40:00.617269 7278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:00.617372 7278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d68790e44f40fdb8ddef524a6d0921, State: Initialized, Role: FOLLOWER
I20260924 10:40:00.617818 7278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d68790e44f40fdb8ddef524a6d0921" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45917 } }
I20260924 10:40:00.617988 7278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:40:00.618083 7278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:40:00.618184 7278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:00.619166 7278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d68790e44f40fdb8ddef524a6d0921" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45917 } }
I20260924 10:40:00.619516 7278 leader_election.cc:302] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95d68790e44f40fdb8ddef524a6d0921; no voters:
I20260924 10:40:00.619855 7278 leader_election.cc:288] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:40:00.619978 7283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:00.620231 7283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [term 1 LEADER]: Becoming Leader. State: Replica: 95d68790e44f40fdb8ddef524a6d0921, State: Running, Role: LEADER
I20260924 10:40:00.620522 7283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d68790e44f40fdb8ddef524a6d0921" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45917 } }
I20260924 10:40:00.621203 7278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:40:00.622236 7285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95d68790e44f40fdb8ddef524a6d0921. Latest consensus state: current_term: 1 leader_uuid: "95d68790e44f40fdb8ddef524a6d0921" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d68790e44f40fdb8ddef524a6d0921" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45917 } } }
I20260924 10:40:00.622406 7285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:00.622766 7284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95d68790e44f40fdb8ddef524a6d0921" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d68790e44f40fdb8ddef524a6d0921" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45917 } } }
I20260924 10:40:00.622866 7284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:00.623857 7292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:40:00.626765 7292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:40:00.632036 7292 catalog_manager.cc:1347] Generated new cluster ID: 06a72ca270a94da18c6e75a110d59cbb
I20260924 10:40:00.632102 7292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:40:00.647697 7292 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:40:00.648629 7292 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:40:00.656684 7292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95d68790e44f40fdb8ddef524a6d0921: Generated new TSK 0
I20260924 10:40:00.657440 7292 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 10:40:00.701061 7308 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 10:40:00.701189 7308 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 10:40:00.701689 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.677177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.2.8.62:45917)
0924 10:40:00.677325 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:00.677330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:00.677629 (+ 299us) server_negotiation.cc:407] Connection header received
0924 10:40:00.677864 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:00.677867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:00.677920 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:00.678005 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:40:00.678320 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:00.678877 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:00.680145 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:00.680371 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:00.680452 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:00.680826 (+ 374us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:40:00.701208 (+ 20382us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 10:40:00.701577 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37266: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 10:40:00.701788 7281 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37266: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 10:40:00.703101 7307 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:00.677105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45917 (local address 127.0.0.1:37266)
0924 10:40:00.677529 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:00.677545 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:00.677683 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:00.678053 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:00.678056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:00.678065 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:40:00.678196 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:00.678202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:00.678990 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:00.678994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:00.680026 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:00.680037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:00.680459 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:00.680482 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:00.680577 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:00.680638 (+ 61us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:40:00.701543 (+ 20905us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 10:40:00.701798 (+ 255us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.8.62:45917: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":116,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.8.62:45917: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 10:40:00.703826 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 7211
2026-09-24T10:40:00Z chronyd exiting
I20260924 10:40:00.741155 2080 test_util.cc:175] -----------------------------------------------
I20260924 10:40:00.741242 2080 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790246374820021-2080-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (357 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 10:40:00.742311 2080 mini_oidc.cc:152] Starting JWKS server
I20260924 10:40:00.743180 2080 webserver.cc:460] Webserver started at https://127.0.0.1:36507/ using document root <none> and password file <none>
I20260924 10:40:00.743294 2080 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 10:40:00.743573 2080 webserver.cc:460] Webserver started at http://127.0.0.1:41223/ using document root <none> and password file <none>
2026-09-24T10:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:40:00Z Disabled control of system clock
I20260924 10:40:00.752264 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:41359
--webserver_interface=127.2.8.62
--webserver_port=0
--builtin_ntp_servers=127.2.8.20:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:41359
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36507/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/testchainca.cert with env {}
W20260924 10:40:00.865589 7314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:00.865918 7314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:00.866009 7314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:00.870215 7314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:40:00.870316 7314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:00.870347 7314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:00.870411 7314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:40:00.870452 7314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:40:00.870496 7314 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:40:00.870535 7314 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36507/jwks.json
I20260924 10:40:00.876034 7314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:41359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:41359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36507/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:00.877426 7314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:00.878528 7314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:00.887244 7320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:00.887277 7322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:00.887284 7319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:00.887794 7314 server_base.cc:1034] running on GCE node
I20260924 10:40:00.888386 7314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:00.889688 7314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:00.890913 7314 hybrid_clock.cc:648] HybridClock initialized: now 1790246400890892 us; error 32 us; skew 500 ppm
I20260924 10:40:00.892800 7314 webserver.cc:460] Webserver started at http://127.2.8.62:33003/ using document root <none> and password file <none>
I20260924 10:40:00.893481 7314 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:00.893580 7314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:00.893819 7314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:00.895855 7314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "6a19ad2fff86418aba5b44fe14200161"
format_stamp: "Formatted at 2026-09-24 10:40:00 on dist-test-slave-j38h"
server_key: "81cc0f119aa1b1ed3a39011eac36893d"
server_key_iv: "233e8cbf4bd9ee2a03a062da14c99a25"
server_key_version: "encryptionkey@0"
I20260924 10:40:00.896422 7314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "6a19ad2fff86418aba5b44fe14200161"
format_stamp: "Formatted at 2026-09-24 10:40:00 on dist-test-slave-j38h"
server_key: "81cc0f119aa1b1ed3a39011eac36893d"
server_key_iv: "233e8cbf4bd9ee2a03a062da14c99a25"
server_key_version: "encryptionkey@0"
I20260924 10:40:00.900046 7314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:40:00.902462 7328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:00.903616 7314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:40:00.903749 7314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "6a19ad2fff86418aba5b44fe14200161"
format_stamp: "Formatted at 2026-09-24 10:40:00 on dist-test-slave-j38h"
server_key: "81cc0f119aa1b1ed3a39011eac36893d"
server_key_iv: "233e8cbf4bd9ee2a03a062da14c99a25"
server_key_version: "encryptionkey@0"
I20260924 10:40:00.903872 7314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:00.920588 7314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:00.921386 7314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:00.921599 7314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:00.928403 7314 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:41359
I20260924 10:40:00.928417 7380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:41359 every 8 connection(s)
I20260924 10:40:00.929483 7314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:40:00.932869 7381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:00.937969 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 7314
I20260924 10:40:00.938124 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:40:00.938388 2080 external_mini_cluster.cc:1428] Setting key abe6253bb08b9bc710132b34861ca317
I20260924 10:40:00.939226 7381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161: Bootstrap starting.
I20260924 10:40:00.941710 7381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:00.942682 7381 log.cc:824] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:00.945129 7381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161: No bootstrap required, opened a new log
I20260924 10:40:00.948871 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.939810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.2.8.62:41359)
0924 10:40:00.940456 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:00.940472 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:00.940508 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:40:00.941238 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:00.941259 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:00.941666 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:00.942004 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:00.942369 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:00.943570 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:00.944877 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:00.945487 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:00.945611 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:00.946586 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:00.946591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:00.946606 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:00.947031 (+ 425us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:00.947091 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:00.947112 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:00.947198 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:00.947375 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 10:40:00.947707 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":152,"threads_started":1}
I20260924 10:40:00.948975 7381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a19ad2fff86418aba5b44fe14200161" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41359 } }
I20260924 10:40:00.949352 7381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:00.949417 7381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a19ad2fff86418aba5b44fe14200161, State: Initialized, Role: FOLLOWER
I20260924 10:40:00.949906 7381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a19ad2fff86418aba5b44fe14200161" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41359 } }
I20260924 10:40:00.950068 7381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:40:00.950132 7381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:40:00.950201 7381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:00.951323 7381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a19ad2fff86418aba5b44fe14200161" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41359 } }
I20260924 10:40:00.951668 7381 leader_election.cc:302] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a19ad2fff86418aba5b44fe14200161; no voters:
I20260924 10:40:00.952024 7381 leader_election.cc:288] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:40:00.952016 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:40:00.952119 7386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:00.952332 7386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [term 1 LEADER]: Becoming Leader. State: Replica: 6a19ad2fff86418aba5b44fe14200161, State: Running, Role: LEADER
I20260924 10:40:00.952684 7386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a19ad2fff86418aba5b44fe14200161" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41359 } }
I20260924 10:40:00.953553 7381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:40:00.954032 7386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a19ad2fff86418aba5b44fe14200161. Latest consensus state: current_term: 1 leader_uuid: "6a19ad2fff86418aba5b44fe14200161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a19ad2fff86418aba5b44fe14200161" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41359 } } }
I20260924 10:40:00.954149 7386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:00.954461 7387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a19ad2fff86418aba5b44fe14200161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a19ad2fff86418aba5b44fe14200161" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41359 } } }
I20260924 10:40:00.954562 7390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:40:00.954588 7387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:00.957795 7390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:40:00.964080 7390 catalog_manager.cc:1347] Generated new cluster ID: d898f18d360b4d2fa31ed546266a775f
I20260924 10:40:00.964178 7390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:40:00.970129 7390 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:40:00.971132 7390 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:40:00.978161 7390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a19ad2fff86418aba5b44fe14200161: Generated new TSK 0
I20260924 10:40:00.978914 7390 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 10:40:01.007014 7411 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 10:40:01.007119 7411 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 10:40:01.007936 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:00.990736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34696 (local address 127.2.8.62:41359)
0924 10:40:00.990877 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:00.990882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:00.991080 (+ 198us) server_negotiation.cc:407] Connection header received
0924 10:40:00.991296 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:00.991299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:00.991349 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:00.991471 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:40:00.991736 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:00.992329 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:00.993440 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:00.993691 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:00.993814 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:00.994148 (+ 334us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:40:01.007417 (+ 13269us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 10:40:01.007705 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34696: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 10:40:01.008014 7384 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34696: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 10:40:01.008083 7410 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:00.990661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41359 (local address 127.0.0.1:34696)
0924 10:40:00.990986 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:00.990999 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:00.991134 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:00.991471 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:00.991475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:00.991484 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:40:00.991611 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:00.991623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:00.992499 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:00.992503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:00.993305 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:00.993314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:00.993802 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:00.993826 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:00.993931 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:00.993955 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:40:01.007682 (+ 13727us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 10:40:01.007893 (+ 211us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.8.62:41359: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36507/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":131,"threads_started":1}
I20260924 10:40:01.008682 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 7314
2026-09-24T10:40:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (305 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 10:40:01.047930 2080 mini_oidc.cc:152] Starting JWKS server
I20260924 10:40:01.048849 2080 webserver.cc:460] Webserver started at https://127.0.0.1:41015/ using document root <none> and password file <none>
I20260924 10:40:01.048966 2080 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 10:40:01.049278 2080 webserver.cc:460] Webserver started at http://127.0.0.1:40117/ using document root <none> and password file <none>
2026-09-24T10:40:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:40:01Z Disabled control of system clock
I20260924 10:40:01.057765 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:42237
--webserver_interface=127.2.8.62
--webserver_port=0
--builtin_ntp_servers=127.2.8.20:41655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:42237
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41015/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 10:40:01.167270 7417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:01.167603 7417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:01.167671 7417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:01.171694 7417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:40:01.171800 7417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:01.171831 7417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:01.171872 7417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:40:01.171913 7417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:40:01.171952 7417 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:40:01.171981 7417 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41015/jwks.json
I20260924 10:40:01.176667 7417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:41655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:42237
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:42237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41015/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:01.177763 7417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:01.179163 7417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:01.187402 7422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:01.187415 7425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:01.187415 7423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:01.187649 7417 server_base.cc:1034] running on GCE node
I20260924 10:40:01.188277 7417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:01.189452 7417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:01.190644 7417 hybrid_clock.cc:648] HybridClock initialized: now 1790246401190624 us; error 32 us; skew 500 ppm
I20260924 10:40:01.192530 7417 webserver.cc:460] Webserver started at http://127.2.8.62:40999/ using document root <none> and password file <none>
I20260924 10:40:01.193192 7417 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:01.193284 7417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:01.193523 7417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:01.195571 7417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "0e9eb2f5c9a54cb699c08fbfc428d7a7"
format_stamp: "Formatted at 2026-09-24 10:40:01 on dist-test-slave-j38h"
server_key: "0609606ee79d607589d8e347c2a49cce"
server_key_iv: "e9a11f96199cf661e66d40bd8308cffb"
server_key_version: "encryptionkey@0"
I20260924 10:40:01.196130 7417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "0e9eb2f5c9a54cb699c08fbfc428d7a7"
format_stamp: "Formatted at 2026-09-24 10:40:01 on dist-test-slave-j38h"
server_key: "0609606ee79d607589d8e347c2a49cce"
server_key_iv: "e9a11f96199cf661e66d40bd8308cffb"
server_key_version: "encryptionkey@0"
I20260924 10:40:01.199720 7417 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 10:40:01.202095 7431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:01.203310 7417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:40:01.203451 7417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "0e9eb2f5c9a54cb699c08fbfc428d7a7"
format_stamp: "Formatted at 2026-09-24 10:40:01 on dist-test-slave-j38h"
server_key: "0609606ee79d607589d8e347c2a49cce"
server_key_iv: "e9a11f96199cf661e66d40bd8308cffb"
server_key_version: "encryptionkey@0"
I20260924 10:40:01.203562 7417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:01.223425 7417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:01.224145 7417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:01.224354 7417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:01.231120 7417 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:42237
I20260924 10:40:01.231125 7483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:42237 every 8 connection(s)
I20260924 10:40:01.232079 7417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:40:01.233016 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 7417
I20260924 10:40:01.233141 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:40:01.233433 2080 external_mini_cluster.cc:1428] Setting key 2c234a44cdb74a5fa3f2c96de88eb6e4
I20260924 10:40:01.235863 7484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:01.241325 7484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7: Bootstrap starting.
I20260924 10:40:01.242678 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:01.234761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.2.8.62:42237)
0924 10:40:01.235412 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:01.235427 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:01.235471 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:40:01.236204 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:01.236221 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:01.236618 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:01.237006 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:01.237437 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:01.238400 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:01.239281 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:01.239756 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:01.239922 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:01.240841 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:01.240846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:01.240858 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:01.241314 (+ 456us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:01.241368 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:01.241388 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:01.241500 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:01.241657 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 10:40:01.241954 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":148,"threads_started":1}
I20260924 10:40:01.243538 7484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:01.244264 7484 log.cc:824] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:01.245420 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:40:01.246376 7484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7: No bootstrap required, opened a new log
I20260924 10:40:01.249974 7484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9eb2f5c9a54cb699c08fbfc428d7a7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42237 } }
I20260924 10:40:01.250216 7484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:01.250279 7484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e9eb2f5c9a54cb699c08fbfc428d7a7, State: Initialized, Role: FOLLOWER
I20260924 10:40:01.250697 7484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9eb2f5c9a54cb699c08fbfc428d7a7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42237 } }
I20260924 10:40:01.250849 7484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:40:01.250934 7484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:40:01.251053 7484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:01.252182 7484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9eb2f5c9a54cb699c08fbfc428d7a7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42237 } }
I20260924 10:40:01.252525 7484 leader_election.cc:302] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e9eb2f5c9a54cb699c08fbfc428d7a7; no voters:
I20260924 10:40:01.252830 7484 leader_election.cc:288] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:40:01.253008 7489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:01.253249 7489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [term 1 LEADER]: Becoming Leader. State: Replica: 0e9eb2f5c9a54cb699c08fbfc428d7a7, State: Running, Role: LEADER
I20260924 10:40:01.253579 7489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9eb2f5c9a54cb699c08fbfc428d7a7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42237 } }
I20260924 10:40:01.254132 7484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:40:01.255256 7491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e9eb2f5c9a54cb699c08fbfc428d7a7. Latest consensus state: current_term: 1 leader_uuid: "0e9eb2f5c9a54cb699c08fbfc428d7a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9eb2f5c9a54cb699c08fbfc428d7a7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42237 } } }
I20260924 10:40:01.255267 7490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e9eb2f5c9a54cb699c08fbfc428d7a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9eb2f5c9a54cb699c08fbfc428d7a7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42237 } } }
I20260924 10:40:01.255402 7490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:01.255401 7491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:01.255726 7498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:40:01.258876 7498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:40:01.264752 7498 catalog_manager.cc:1347] Generated new cluster ID: b1485bab1dd5448fadef24699ca2c21b
I20260924 10:40:01.264833 7498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:40:01.278024 7498 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:40:01.278991 7498 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:40:01.292784 7498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e9eb2f5c9a54cb699c08fbfc428d7a7: Generated new TSK 0
I20260924 10:40:01.293624 7498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:40:01.318111 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:01.311947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.2.8.62:42237)
0924 10:40:01.312088 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:01.312092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:01.312209 (+ 117us) server_negotiation.cc:407] Connection header received
0924 10:40:01.312350 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:01.312353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:01.312415 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:01.312561 (+ 146us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:40:01.312756 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:01.313685 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:01.314545 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:01.314916 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:01.315037 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:01.315380 (+ 343us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:40:01.317704 (+ 2324us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41015/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 10:40:01.317958 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37054: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41015/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 10:40:01.318188 7487 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37054: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41015/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 10:40:01.319582 7513 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:01.311834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:42237 (local address 127.0.0.1:37054)
0924 10:40:01.312105 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:01.312123 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:01.312220 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:01.312548 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:01.312550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:01.312556 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:40:01.312640 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:01.312644 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:01.313795 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:01.313797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:01.314432 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:01.314439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:01.315053 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:01.315077 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:01.315147 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:01.315172 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:40:01.317879 (+ 2707us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41015/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 10:40:01.319413 (+ 1534us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.8.62:42237: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41015/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":1355,"thread_start_us":102,"threads_started":1}
I20260924 10:40:01.320135 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 7417
E20260924 10:40:01.328738 7514 webserver.cc:533] Webserver: sslize failed:
E20260924 10:40:01.328832 7514 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T10:40:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (311 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T10:40:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:40:01Z Disabled control of system clock
I20260924 10:40:01.368507 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:36789
--webserver_interface=127.2.8.62
--webserver_port=0
--builtin_ntp_servers=127.2.8.20:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:36789
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/testchainca.cert with env {}
W20260924 10:40:01.477766 7518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:01.478103 7518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:01.478193 7518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:01.482251 7518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:40:01.482358 7518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:01.482388 7518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:01.482427 7518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:40:01.482465 7518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 10:40:01.482506 7518 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 10:40:01.482537 7518 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 10:40:01.487905 7518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:36789
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:36789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:01.489061 7518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:01.490096 7518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:01.498817 7523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:01.498828 7524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:01.498857 7526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:01.499176 7518 server_base.cc:1034] running on GCE node
I20260924 10:40:01.499702 7518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:01.500872 7518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:01.502102 7518 hybrid_clock.cc:648] HybridClock initialized: now 1790246401502104 us; error 53 us; skew 500 ppm
I20260924 10:40:01.504071 7518 webserver.cc:460] Webserver started at http://127.2.8.62:39137/ using document root <none> and password file <none>
I20260924 10:40:01.504652 7518 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:01.504741 7518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:01.505015 7518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:01.507121 7518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "1d2818a030934a6b999aa53026c36eaa"
format_stamp: "Formatted at 2026-09-24 10:40:01 on dist-test-slave-j38h"
server_key: "04bdff4e0aab5cff348e68d987361ff8"
server_key_iv: "098f48427475001788b4392027142042"
server_key_version: "encryptionkey@0"
I20260924 10:40:01.507683 7518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "1d2818a030934a6b999aa53026c36eaa"
format_stamp: "Formatted at 2026-09-24 10:40:01 on dist-test-slave-j38h"
server_key: "04bdff4e0aab5cff348e68d987361ff8"
server_key_iv: "098f48427475001788b4392027142042"
server_key_version: "encryptionkey@0"
I20260924 10:40:01.511502 7518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:40:01.514520 7532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:01.515714 7518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:40:01.515821 7518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "1d2818a030934a6b999aa53026c36eaa"
format_stamp: "Formatted at 2026-09-24 10:40:01 on dist-test-slave-j38h"
server_key: "04bdff4e0aab5cff348e68d987361ff8"
server_key_iv: "098f48427475001788b4392027142042"
server_key_version: "encryptionkey@0"
I20260924 10:40:01.515908 7518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:01.529745 7518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:01.530395 7518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:01.530553 7518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:01.537361 7518 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:36789
I20260924 10:40:01.537385 7584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:36789 every 8 connection(s)
I20260924 10:40:01.538331 7518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:40:01.541484 7585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:01.544186 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 7518
I20260924 10:40:01.544299 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:40:01.544600 2080 external_mini_cluster.cc:1428] Setting key 2e97d564208176d51ea442f3ad1c35d2
I20260924 10:40:01.547621 7585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa: Bootstrap starting.
I20260924 10:40:01.550377 7585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:01.551153 7585 log.cc:824] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:01.553365 7585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa: No bootstrap required, opened a new log
I20260924 10:40:01.555274 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:01.546075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43998 (local address 127.2.8.62:36789)
0924 10:40:01.546797 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:01.546819 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:01.546861 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:40:01.547657 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:01.547679 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:01.548181 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:01.548562 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:01.549068 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:01.550509 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:01.551507 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:01.552098 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:01.552254 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:01.553182 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:01.553188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:01.553204 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:01.553727 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:01.553796 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:01.553821 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:01.553922 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:01.554095 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 10:40:01.554360 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":117,"threads_started":1}
I20260924 10:40:01.557812 7585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2818a030934a6b999aa53026c36eaa" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 36789 } }
I20260924 10:40:01.558071 7585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:01.558125 7585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2818a030934a6b999aa53026c36eaa, State: Initialized, Role: FOLLOWER
I20260924 10:40:01.558249 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:40:01.558610 7585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2818a030934a6b999aa53026c36eaa" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 36789 } }
I20260924 10:40:01.558773 7585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:40:01.558832 7585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:40:01.558918 7585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:01.560058 7585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2818a030934a6b999aa53026c36eaa" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 36789 } }
I20260924 10:40:01.560395 7585 leader_election.cc:302] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d2818a030934a6b999aa53026c36eaa; no voters:
I20260924 10:40:01.560746 7585 leader_election.cc:288] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:40:01.560917 7590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:01.561200 7590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [term 1 LEADER]: Becoming Leader. State: Replica: 1d2818a030934a6b999aa53026c36eaa, State: Running, Role: LEADER
I20260924 10:40:01.561512 7590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2818a030934a6b999aa53026c36eaa" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 36789 } }
I20260924 10:40:01.562057 7585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:40:01.563331 7591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d2818a030934a6b999aa53026c36eaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2818a030934a6b999aa53026c36eaa" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 36789 } } }
I20260924 10:40:01.563499 7591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:01.563335 7592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d2818a030934a6b999aa53026c36eaa. Latest consensus state: current_term: 1 leader_uuid: "1d2818a030934a6b999aa53026c36eaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2818a030934a6b999aa53026c36eaa" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 36789 } } }
I20260924 10:40:01.563817 7592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa [sys.catalog]: This master's current role is: LEADER
W20260924 10:40:01.566728 7605 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:40:01.566812 7605 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:40:01.567149 7600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:40:01.567824 7600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:40:01.573352 7600 catalog_manager.cc:1347] Generated new cluster ID: c2206a33d475477b9c6da47a0dfd6be0
I20260924 10:40:01.573433 7600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:40:01.580698 7600 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:40:01.581679 7600 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:40:01.590662 7600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d2818a030934a6b999aa53026c36eaa: Generated new TSK 0
I20260924 10:40:01.591482 7600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:40:01.654199 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:01.606443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44012 (local address 127.2.8.62:36789)
0924 10:40:01.606588 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:01.606592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:01.606723 (+ 131us) server_negotiation.cc:407] Connection header received
0924 10:40:01.606901 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:01.606904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:01.606968 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:01.607109 (+ 141us) server_negotiation.cc:226] Negotiated authn=JWT
0924 10:40:01.607304 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:01.608144 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:01.609054 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:01.609405 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:01.609521 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:01.609805 (+ 284us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 10:40:01.653706 (+ 43901us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 10:40:01.654040 (+ 334us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44012: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 10:40:01.654289 7588 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44012: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 10:40:01.654510 7614 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:01.606343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:36789 (local address 127.0.0.1:44012)
0924 10:40:01.606620 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:01.606635 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:01.606758 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:01.607101 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:01.607103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:01.607110 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0924 10:40:01.607200 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:01.607204 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:01.608258 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:01.608261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:01.608891 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:01.608899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:01.609507 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:01.609528 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:01.609595 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:01.609617 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 10:40:01.653997 (+ 44380us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 10:40:01.654181 (+ 184us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.8.62:36789: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":240,"threads_started":1}
I20260924 10:40:01.655095 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 7518
2026-09-24T10:40:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (314 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (127 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:40:01 dist-test-slave-j38h krb5kdc[7626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:40:01 dist-test-slave-j38h krb5kdc[7626](info): set up 2 sockets
Sep 24 10:40:01 dist-test-slave-j38h krb5kdc[7626](info): commencing operation
krb5kdc: starting...
W20260924 10:40:02.941593 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:40:03 dist-test-slave-j38h krb5kdc[7626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:40:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:40:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:40:03.068246 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:46567
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:46567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:03.176826 7639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:03.177181 7639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:03.177265 7639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:03.181435 7639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:40:03.181547 7639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:03.181582 7639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:03.181622 7639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:40:03.181654 7639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:40:03.187044 7639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:46567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:46567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:03.188195 7639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:03.189276 7639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:03.197834 7644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:03.198084 7639 server_base.cc:1034] running on GCE node
W20260924 10:40:03.197834 7645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:03.197852 7647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:03.198921 7639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:03.200103 7639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:03.201294 7639 hybrid_clock.cc:648] HybridClock initialized: now 1790246403201279 us; error 33 us; skew 500 ppm
Sep 24 10:40:03 dist-test-slave-j38h krb5kdc[7626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:03.204438 7639 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:40:03.205569 7639 webserver.cc:460] Webserver started at http://127.2.8.62:38101/ using document root <none> and password file <none>
I20260924 10:40:03.206213 7639 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:03.206321 7639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:03.206578 7639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:03.209038 7639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "8680a4e5679f41b8a6717e1598b03a9d"
format_stamp: "Formatted at 2026-09-24 10:40:03 on dist-test-slave-j38h"
server_key: "509497250c92f6a0f5720817fc83ee06"
server_key_iv: "de8890623a6380ca3d519e96849156bd"
server_key_version: "encryptionkey@0"
I20260924 10:40:03.209690 7639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "8680a4e5679f41b8a6717e1598b03a9d"
format_stamp: "Formatted at 2026-09-24 10:40:03 on dist-test-slave-j38h"
server_key: "509497250c92f6a0f5720817fc83ee06"
server_key_iv: "de8890623a6380ca3d519e96849156bd"
server_key_version: "encryptionkey@0"
I20260924 10:40:03.214033 7639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:40:03.216747 7654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:03.218209 7639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:40:03.218366 7639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "8680a4e5679f41b8a6717e1598b03a9d"
format_stamp: "Formatted at 2026-09-24 10:40:03 on dist-test-slave-j38h"
server_key: "509497250c92f6a0f5720817fc83ee06"
server_key_iv: "de8890623a6380ca3d519e96849156bd"
server_key_version: "encryptionkey@0"
I20260924 10:40:03.218550 7639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:03.237973 7639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:03.241112 7639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:03.241324 7639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:03.248663 7639 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:46567
I20260924 10:40:03.248678 7706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:46567 every 8 connection(s)
I20260924 10:40:03.249641 7639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:40:03.252547 7707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:03.255049 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 7639
I20260924 10:40:03.255193 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:40:03.255517 2080 external_mini_cluster.cc:1428] Setting key 7abebd0f26b8dc8adf58223dd6a9c42c
I20260924 10:40:03.259073 7707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d: Bootstrap starting.
I20260924 10:40:03.261433 7707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:03.262200 7707 log.cc:824] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:40:03 dist-test-slave-j38h krb5kdc[7626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:40:03.264513 7707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d: No bootstrap required, opened a new log
I20260924 10:40:03.267854 7707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8680a4e5679f41b8a6717e1598b03a9d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46567 } }
I20260924 10:40:03.268115 7707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:03.268204 7707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8680a4e5679f41b8a6717e1598b03a9d, State: Initialized, Role: FOLLOWER
I20260924 10:40:03.268739 7707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8680a4e5679f41b8a6717e1598b03a9d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46567 } }
I20260924 10:40:03.268911 7707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:40:03.269014 7707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:40:03.269127 7707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:03.270218 7707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8680a4e5679f41b8a6717e1598b03a9d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46567 } }
I20260924 10:40:03.270514 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:03.257138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42948 (local address 127.2.8.62:46567)
0924 10:40:03.257569 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:03.257584 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:03.257618 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:40:03.258235 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:03.258251 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:03.258582 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:03.258922 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:03.259691 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:03.260677 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:03.261646 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:03.262152 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:03.262336 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:03.265155 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:03.265161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:03.265175 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:03.265200 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:03.267424 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:03.267870 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:03.267881 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:03.267885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:03.267995 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:03.268316 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:03.268320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:03.268321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:03.268844 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:03.269102 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:03.269429 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 10:40:03.269725 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":138,"threads_started":1}
I20260924 10:40:03.270592 7707 leader_election.cc:302] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8680a4e5679f41b8a6717e1598b03a9d; no voters:
I20260924 10:40:03.270989 7707 leader_election.cc:288] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:40:03.271067 7712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:03.271283 7712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [term 1 LEADER]: Becoming Leader. State: Replica: 8680a4e5679f41b8a6717e1598b03a9d, State: Running, Role: LEADER
I20260924 10:40:03.271642 7712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8680a4e5679f41b8a6717e1598b03a9d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46567 } }
I20260924 10:40:03.273274 7713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8680a4e5679f41b8a6717e1598b03a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8680a4e5679f41b8a6717e1598b03a9d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46567 } } }
I20260924 10:40:03.273440 7713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:03.273540 7707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:40:03.273693 7713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8680a4e5679f41b8a6717e1598b03a9d. Latest consensus state: current_term: 1 leader_uuid: "8680a4e5679f41b8a6717e1598b03a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8680a4e5679f41b8a6717e1598b03a9d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46567 } } }
I20260924 10:40:03.273787 7713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d [sys.catalog]: This master's current role is: LEADER
W20260924 10:40:03.278088 7728 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:40:03.278157 7728 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:40:03.278234 7716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:40:03.278872 7716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:40:03.283840 7716 catalog_manager.cc:1347] Generated new cluster ID: 1a9185c726234de5addd87b81f771f8a
I20260924 10:40:03.283910 7716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:40:03.309237 7716 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:40:03.310648 7716 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:40:03.339799 7716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8680a4e5679f41b8a6717e1598b03a9d: Generated new TSK 0
I20260924 10:40:03.340600 7716 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:40:03.375749 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:46567
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:03.488198 7733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:03.488492 7733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:03.488536 7733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:03.492662 7733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:03.492739 7733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:03.492822 7733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:40:03.498245 7733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:46567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:03.499251 7733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:03.500252 7733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:03.510094 7739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:03.510154 7733 server_base.cc:1034] running on GCE node
W20260924 10:40:03.510115 7741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:03.510094 7738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:03.510784 7733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:03.511533 7733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:03.512701 7733 hybrid_clock.cc:648] HybridClock initialized: now 1790246403512681 us; error 31 us; skew 500 ppm
Sep 24 10:40:03 dist-test-slave-j38h krb5kdc[7626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:03.528132 7733 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:40:03.529277 7733 webserver.cc:460] Webserver started at http://127.2.8.1:46875/ using document root <none> and password file <none>
I20260924 10:40:03.529868 7733 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:03.529929 7733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:03.530174 7733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:03.532254 7733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "faba837bab9a4551964f5a4ec54b4c18"
format_stamp: "Formatted at 2026-09-24 10:40:03 on dist-test-slave-j38h"
server_key: "87d8c3db876dfd5b42a6766ec27d3330"
server_key_iv: "d3950076322b0ad4af5f48d156260f43"
server_key_version: "encryptionkey@0"
I20260924 10:40:03.532829 7733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "faba837bab9a4551964f5a4ec54b4c18"
format_stamp: "Formatted at 2026-09-24 10:40:03 on dist-test-slave-j38h"
server_key: "87d8c3db876dfd5b42a6766ec27d3330"
server_key_iv: "d3950076322b0ad4af5f48d156260f43"
server_key_version: "encryptionkey@0"
I20260924 10:40:03.536535 7733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:40:03.539036 7748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:03.540264 7733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:40:03.540385 7733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "faba837bab9a4551964f5a4ec54b4c18"
format_stamp: "Formatted at 2026-09-24 10:40:03 on dist-test-slave-j38h"
server_key: "87d8c3db876dfd5b42a6766ec27d3330"
server_key_iv: "d3950076322b0ad4af5f48d156260f43"
server_key_version: "encryptionkey@0"
I20260924 10:40:03.540493 7733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:03.563377 7733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:03.566871 7733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:03.567097 7733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:03.567720 7733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:03.568562 7733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:03.568609 7733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:03.568677 7733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:03.568708 7733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:03.577798 7733 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:37381
I20260924 10:40:03.577822 7861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:37381 every 8 connection(s)
I20260924 10:40:03.578817 7733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:40:03.582667 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 7733
I20260924 10:40:03.582787 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:40:03.583092 2080 external_mini_cluster.cc:1428] Setting key adf2e9f1ad47d771688c5c44e857191a
Sep 24 10:40:03 dist-test-slave-j38h krb5kdc[7626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
I20260924 10:40:03.593392 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:03.580664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:58155 (local address 127.2.8.62:46567)
0924 10:40:03.580807 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:03.580810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:03.581713 (+ 903us) server_negotiation.cc:407] Connection header received
0924 10:40:03.583295 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:03.583298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:03.583347 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:03.583415 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:03.584847 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:03.585431 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:03.586910 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:03.587151 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:03.587263 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:03.589807 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:03.589810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:03.589812 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:03.589836 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:03.591270 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:03.591772 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:03.591776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:03.591778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:03.591861 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:03.592207 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:03.592211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:03.592212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:03.592434 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:03.592536 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:03.593089 (+ 553us) server_negotiation.cc:299] Negotiation successful
0924 10:40:03.593215 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:40:03.594383 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:03.581073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46567 (local address 127.2.8.1:58155)
0924 10:40:03.581559 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:03.581599 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:03.583053 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:03.583645 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:03.583652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:03.584087 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:03.584698 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:03.584717 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:03.585557 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:03.585564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:03.586783 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:03.586792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:03.587249 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:03.587304 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:03.587478 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:03.588091 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:03.588106 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:03.589634 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:03.591388 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:03.591394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:03.591411 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:03.591669 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:03.591957 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:03.591960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:03.591962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:03.592106 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:03.592544 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:03.592553 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:03.592802 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 10:40:03.593009 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 10:40:03.593344 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":145,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
I20260924 10:40:03.596648 7862 heartbeater.cc:347] Connected to a master server at 127.2.8.62:46567
I20260924 10:40:03.596922 7862 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:03.597435 7862 heartbeater.cc:511] Master 127.2.8.62:46567 requested a full tablet report, sending...
I20260924 10:40:03.599078 7671 ts_manager.cc:194] Registered new tserver with Master: faba837bab9a4551964f5a4ec54b4c18 (127.2.8.1:37381)
I20260924 10:40:03.600780 7671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:58155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:40:03.611855 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:46567
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:03.723521 7867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:03.723783 7867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:03.723827 7867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:03.728088 7867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:03.728154 7867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:03.728246 7867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:40:03.733633 7867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:46567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:03.734711 7867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:03.735777 7867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:03.745453 7875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:03.745447 7872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:03.745575 7873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:03.745647 7867 server_base.cc:1034] running on GCE node
I20260924 10:40:03.746124 7867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:03.746835 7867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:03.748030 7867 hybrid_clock.cc:648] HybridClock initialized: now 1790246403748001 us; error 45 us; skew 500 ppm
Sep 24 10:40:03 dist-test-slave-j38h krb5kdc[7626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:03.750856 7867 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:40:03.751751 7867 webserver.cc:460] Webserver started at http://127.2.8.2:33339/ using document root <none> and password file <none>
I20260924 10:40:03.752321 7867 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:03.752410 7867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:03.752661 7867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:03.754702 7867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "358e35b9fa3b4e13b8f66e290fa2c802"
format_stamp: "Formatted at 2026-09-24 10:40:03 on dist-test-slave-j38h"
server_key: "299247e4f1b5b4dcf101d66c6c0bdd99"
server_key_iv: "e0bd97a02565b2a3517223adb3222c94"
server_key_version: "encryptionkey@0"
I20260924 10:40:03.755271 7867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "358e35b9fa3b4e13b8f66e290fa2c802"
format_stamp: "Formatted at 2026-09-24 10:40:03 on dist-test-slave-j38h"
server_key: "299247e4f1b5b4dcf101d66c6c0bdd99"
server_key_iv: "e0bd97a02565b2a3517223adb3222c94"
server_key_version: "encryptionkey@0"
I20260924 10:40:03.758980 7867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:40:03.761449 7882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:03.762639 7867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:40:03.762771 7867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "358e35b9fa3b4e13b8f66e290fa2c802"
format_stamp: "Formatted at 2026-09-24 10:40:03 on dist-test-slave-j38h"
server_key: "299247e4f1b5b4dcf101d66c6c0bdd99"
server_key_iv: "e0bd97a02565b2a3517223adb3222c94"
server_key_version: "encryptionkey@0"
I20260924 10:40:03.762880 7867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:03.778936 7867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:03.782424 7867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:03.782640 7867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:03.783244 7867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:03.784077 7867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:03.784147 7867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:03.784216 7867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:03.784269 7867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:03.793437 7867 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:35021
I20260924 10:40:03.793457 7995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:35021 every 8 connection(s)
I20260924 10:40:03.794349 7867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:40:03.798390 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 7867
I20260924 10:40:03.798480 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:40:03.798753 2080 external_mini_cluster.cc:1428] Setting key 03b86dcedb9f9ef6db2bfc464621f7b3
Sep 24 10:40:03 dist-test-slave-j38h krb5kdc[7626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
I20260924 10:40:03.809151 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:03.796252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:39517 (local address 127.2.8.62:46567)
0924 10:40:03.796463 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:03.796467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:03.797344 (+ 877us) server_negotiation.cc:407] Connection header received
0924 10:40:03.798833 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:03.798837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:03.798896 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:03.798968 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:03.800348 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:03.800880 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:03.802475 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:03.802687 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:03.802765 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:03.805268 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:03.805275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:03.805280 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:03.805314 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:03.806969 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:03.807475 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:03.807482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:03.807485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:03.807571 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:03.807902 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:03.807909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:03.807911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:03.808183 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:03.808295 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:03.808850 (+ 555us) server_negotiation.cc:299] Negotiation successful
0924 10:40:03.809020 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260924 10:40:03.810077 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:03.796646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46567 (local address 127.2.8.2:39517)
0924 10:40:03.797193 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:03.797237 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:03.798606 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:03.799140 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:03.799147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:03.799613 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:03.800213 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:03.800229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:03.801049 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:03.801053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:03.802326 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:03.802337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:03.802783 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:03.802841 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:03.803027 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:03.803646 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:03.803662 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:03.805103 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:03.807090 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:03.807094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:03.807107 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:03.807349 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:03.807661 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:03.807663 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:03.807665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:03.807797 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:03.808308 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:03.808313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:03.808550 (+ 237us) client_negotiation.cc:769] Sending connection context
0924 10:40:03.808748 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 10:40:03.809073 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
I20260924 10:40:03.811329 7996 heartbeater.cc:347] Connected to a master server at 127.2.8.62:46567
I20260924 10:40:03.811609 7996 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:03.812198 7996 heartbeater.cc:511] Master 127.2.8.62:46567 requested a full tablet report, sending...
I20260924 10:40:03.813480 7671 ts_manager.cc:194] Registered new tserver with Master: 358e35b9fa3b4e13b8f66e290fa2c802 (127.2.8.2:35021)
I20260924 10:40:03.814198 7671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:39517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:40:03.828855 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:46567
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:41031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:03.938558 8001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:03.938858 8001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:03.938936 8001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:03.942970 8001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:03.943058 8001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:03.943145 8001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:40:03.948510 8001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:41031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:46567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:03.949622 8001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:03.950654 8001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:03.960395 8006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:03.960412 8009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:03.960439 8007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:03.960582 8001 server_base.cc:1034] running on GCE node
I20260924 10:40:03.961131 8001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:03.961828 8001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:03.962986 8001 hybrid_clock.cc:648] HybridClock initialized: now 1790246403962951 us; error 51 us; skew 500 ppm
Sep 24 10:40:03 dist-test-slave-j38h krb5kdc[7626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:03.965946 8001 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:40:03.966967 8001 webserver.cc:460] Webserver started at http://127.2.8.3:43983/ using document root <none> and password file <none>
I20260924 10:40:03.967593 8001 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:03.967657 8001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:03.967911 8001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:03.970048 8001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "9d90bbc5555242a78a75c8218051c7ef"
format_stamp: "Formatted at 2026-09-24 10:40:03 on dist-test-slave-j38h"
server_key: "fbf3cd9dd38e9ba06e71c7ad3939f948"
server_key_iv: "16c64ee9935b11489af4f37c1711fd8b"
server_key_version: "encryptionkey@0"
I20260924 10:40:03.970610 8001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "9d90bbc5555242a78a75c8218051c7ef"
format_stamp: "Formatted at 2026-09-24 10:40:03 on dist-test-slave-j38h"
server_key: "fbf3cd9dd38e9ba06e71c7ad3939f948"
server_key_iv: "16c64ee9935b11489af4f37c1711fd8b"
server_key_version: "encryptionkey@0"
I20260924 10:40:03.974373 8001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:40:03.976892 8016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:03.978120 8001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:40:03.978241 8001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "9d90bbc5555242a78a75c8218051c7ef"
format_stamp: "Formatted at 2026-09-24 10:40:03 on dist-test-slave-j38h"
server_key: "fbf3cd9dd38e9ba06e71c7ad3939f948"
server_key_iv: "16c64ee9935b11489af4f37c1711fd8b"
server_key_version: "encryptionkey@0"
I20260924 10:40:03.978348 8001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:03.989792 8001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:03.992592 8001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:03.992798 8001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:03.993469 8001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:03.994295 8001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:03.994339 8001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:03.994407 8001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:03.994448 8001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:04.003276 8001 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:43447
I20260924 10:40:04.003294 8129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:43447 every 8 connection(s)
I20260924 10:40:04.004182 8001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:40:04.005189 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 8001
I20260924 10:40:04.005298 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:40:04.005645 2080 external_mini_cluster.cc:1428] Setting key d1d9e7b7f9a4b18a445bed871313d362
Sep 24 10:40:04 dist-test-slave-j38h krb5kdc[7626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:40:04.018158 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:04.006201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:46349 (local address 127.2.8.62:46567)
0924 10:40:04.006334 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:04.006339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:04.007389 (+ 1050us) server_negotiation.cc:407] Connection header received
0924 10:40:04.008855 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:04.008858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:04.008901 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:04.009020 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:04.010149 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:04.010669 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:04.011747 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:04.011959 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:04.012034 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:04.014651 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:04.014654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:04.014656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:04.014679 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:04.015994 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:04.016502 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:04.016508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:04.016509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:04.016589 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:04.016926 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:04.016930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:04.016932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:04.017227 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:04.017338 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:04.017856 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 10:40:04.017991 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:40:04.019212 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:04.006723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46567 (local address 127.2.8.3:46349)
0924 10:40:04.007221 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:04.007276 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:04.008663 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:04.009177 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:04.009183 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:04.009532 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:04.010028 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:04.010041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:04.010774 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:04.010777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:04.011647 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:04.011655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:04.012040 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:04.012081 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:04.012219 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:04.012848 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:04.012871 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:04.014405 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:04.016120 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:04.016126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:04.016141 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:04.016393 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:04.016684 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:04.016689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:04.016691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:04.016831 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:04.017345 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:04.017354 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:04.017621 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 10:40:04.017806 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 10:40:04.018148 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":115,"threads_started":1}
I20260924 10:40:04.020332 8130 heartbeater.cc:347] Connected to a master server at 127.2.8.62:46567
I20260924 10:40:04.020602 8130 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:04.021147 8130 heartbeater.cc:511] Master 127.2.8.62:46567 requested a full tablet report, sending...
I20260924 10:40:04.022198 7671 ts_manager.cc:194] Registered new tserver with Master: 9d90bbc5555242a78a75c8218051c7ef (127.2.8.3:43447)
I20260924 10:40:04.022753 7671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:46349
I20260924 10:40:04.031529 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/test-admin.keytab.
Sep 24 10:40:04 dist-test-slave-j38h krb5kdc[7626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:04.049443 2080 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 10:40:04.052310 8137 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:04.050847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40002 (local address 127.2.8.1:37381)
0924 10:40:04.051125 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:04.051141 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:04.051169 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:40:04.051265 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:04.051274 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:04.051496 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:04.051671 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:04.052136 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40002: BlockingRecv error: recv got EOF from 127.0.0.1:40002 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
W20260924 10:40:04.052448 2080 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/krb5cc' of type FILE
Sep 24 10:40:04 dist-test-slave-j38h krb5kdc[7626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:04.053570 2080 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 10:40:04 dist-test-slave-j38h krb5kdc[7626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:04.054476 2080 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 10:40:04 dist-test-slave-j38h krb5kdc[7626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.1@KRBTEST.COM
I20260924 10:40:04.064435 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:04.055316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40008 (local address 127.2.8.1:37381)
0924 10:40:04.055455 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:04.055459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:04.055635 (+ 176us) server_negotiation.cc:407] Connection header received
0924 10:40:04.055728 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:04.055731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:04.055771 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:04.055849 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:04.056339 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:04.057250 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:04.058051 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:04.058461 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:04.058551 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:04.060853 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:04.060861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:04.060876 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:04.060899 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:04.062579 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:04.062999 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:04.063005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:04.063006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:04.063091 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:04.063353 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:04.063356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:04.063357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:04.063636 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:04.063805 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:04.064033 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 10:40:04.064234 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 10:40:04.068673 8137 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:04.067599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.2.8.1:37381)
0924 10:40:04.067745 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:04.067750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:04.067871 (+ 121us) server_negotiation.cc:407] Connection header received
0924 10:40:04.068026 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:04.068031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:04.068076 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:04.068161 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:04.068574 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40020: BlockingRecv error: recv got EOF from 127.0.0.1:40020 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 10:40:04.069101 2080 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/krb5cc' of type FILE
Sep 24 10:40:04 dist-test-slave-j38h krb5kdc[7626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:04.070178 2080 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 10:40:04 dist-test-slave-j38h krb5kdc[7626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:04.071120 2080 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 10:40:04 dist-test-slave-j38h krb5kdc[7626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.1@KRBTEST.COM
I20260924 10:40:04.079932 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:04.071919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.2.8.1:37381)
0924 10:40:04.072035 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:04.072042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:04.072149 (+ 107us) server_negotiation.cc:407] Connection header received
0924 10:40:04.072243 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:04.072245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:04.072280 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:04.072346 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:04.072821 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:04.073486 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:04.074449 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:04.074684 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:04.074799 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:04.077077 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:04.077081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:04.077083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:04.077106 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:04.078310 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:04.078736 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:04.078741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:04.078743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:04.078828 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:04.079103 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:04.079106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:04.079108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:04.079273 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:04.079405 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:04.079635 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 10:40:04.079745 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 10:40:04.082710 8137 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:04.081733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40040 (local address 127.2.8.1:37381)
0924 10:40:04.081887 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:04.081893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:04.082007 (+ 114us) server_negotiation.cc:407] Connection header received
0924 10:40:04.082099 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:04.082101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:04.082133 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:04.082194 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:04.082600 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40040: BlockingRecv error: recv got EOF from 127.0.0.1:40040 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260924 10:40:04.082779 2080 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790246374820021-2080-0/krb5kdc/krb5cc' of type FILE
Sep 24 10:40:04 dist-test-slave-j38h krb5kdc[7626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:04.083889 2080 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 10:40:04 dist-test-slave-j38h krb5kdc[7626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:04.084772 2080 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 10:40:04 dist-test-slave-j38h krb5kdc[7626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.1@KRBTEST.COM
I20260924 10:40:04.093116 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:04.085668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40044 (local address 127.2.8.1:37381)
0924 10:40:04.085791 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:04.085795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:04.085895 (+ 100us) server_negotiation.cc:407] Connection header received
0924 10:40:04.086040 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:04.086043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:04.086086 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:04.086167 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:04.086703 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:04.087197 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:04.087969 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:04.088187 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:04.088288 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:04.090474 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:04.090477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:04.090479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:04.090500 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:04.091751 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:04.092112 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:04.092115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:04.092116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:04.092184 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:04.092406 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:04.092409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:04.092411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:04.092561 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:04.092654 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:04.092834 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 10:40:04.092927 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:40:04.094106 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 7733
I20260924 10:40:04.099596 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 7867
I20260924 10:40:04.104604 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 8001
I20260924 10:40:04.111328 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 7639
2026-09-24T10:40:04Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2210 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:40:04 dist-test-slave-j38h krb5kdc[8150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:40:04 dist-test-slave-j38h krb5kdc[8150](info): set up 2 sockets
Sep 24 10:40:04 dist-test-slave-j38h krb5kdc[8150](info): commencing operation
krb5kdc: starting...
W20260924 10:40:05.161481 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:40:05 dist-test-slave-j38h krb5kdc[8150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:40:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:40:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.62.keytab.
Entry for principal oryx/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.62.keytab.
I20260924 10:40:05.292064 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:38611
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.62.keytab
--principal=oryx/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:38611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:05.412135 8163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:05.412396 8163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:05.412436 8163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:05.416402 8163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:40:05.416496 8163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:05.416517 8163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:05.416533 8163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:40:05.416548 8163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:40:05.421926 8163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:38611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.62.keytab
--principal=oryx/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:38611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:05.422912 8163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:05.423861 8163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:05.432030 8169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:05.432053 8168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:05.432053 8171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:05.432101 8163 server_base.cc:1034] running on GCE node
I20260924 10:40:05.433105 8163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:05.434201 8163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:05.435364 8163 hybrid_clock.cc:648] HybridClock initialized: now 1790246405435349 us; error 27 us; skew 500 ppm
Sep 24 10:40:05 dist-test-slave-j38h krb5kdc[8150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246405, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:05.438215 8163 init.cc:377] Logged in from keytab as oryx/127.2.8.62@KRBTEST.COM (short username oryx)
I20260924 10:40:05.439102 8163 webserver.cc:460] Webserver started at http://127.2.8.62:33409/ using document root <none> and password file <none>
I20260924 10:40:05.439672 8163 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:05.439764 8163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:05.439998 8163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:05.442104 8163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "ee68668fd32841e2a0495725f909eee7"
format_stamp: "Formatted at 2026-09-24 10:40:05 on dist-test-slave-j38h"
server_key: "1c100e6047552c99ddb89663eea10e75"
server_key_iv: "3f4023aff97666b0d5ee40826aadad04"
server_key_version: "encryptionkey@0"
I20260924 10:40:05.442683 8163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "ee68668fd32841e2a0495725f909eee7"
format_stamp: "Formatted at 2026-09-24 10:40:05 on dist-test-slave-j38h"
server_key: "1c100e6047552c99ddb89663eea10e75"
server_key_iv: "3f4023aff97666b0d5ee40826aadad04"
server_key_version: "encryptionkey@0"
I20260924 10:40:05.446341 8163 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:40:05.448818 8178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:05.450132 8163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:40:05.450318 8163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "ee68668fd32841e2a0495725f909eee7"
format_stamp: "Formatted at 2026-09-24 10:40:05 on dist-test-slave-j38h"
server_key: "1c100e6047552c99ddb89663eea10e75"
server_key_iv: "3f4023aff97666b0d5ee40826aadad04"
server_key_version: "encryptionkey@0"
I20260924 10:40:05.450438 8163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:05.468912 8163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:05.472208 8163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:05.472441 8163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:05.479306 8163 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:38611
I20260924 10:40:05.479622 8230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:38611 every 8 connection(s)
I20260924 10:40:05.480300 8163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:40:05.483134 8231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:05.488440 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 8163
I20260924 10:40:05.488551 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:40:05.488809 2080 external_mini_cluster.cc:1428] Setting key 363a244a6d7f06b3f792bc49c48b245f
I20260924 10:40:05.489130 8231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7: Bootstrap starting.
I20260924 10:40:05.491899 8231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:05.492659 8231 log.cc:824] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:05.494498 8231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7: No bootstrap required, opened a new log
I20260924 10:40:05.498631 8231 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } }
I20260924 10:40:05.498879 8231 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:05.498946 8231 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee68668fd32841e2a0495725f909eee7, State: Initialized, Role: FOLLOWER
Sep 24 10:40:05 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM
I20260924 10:40:05.499404 8231 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } }
I20260924 10:40:05.499553 8231 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:40:05.499615 8231 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:40:05.499715 8231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:05.500916 8231 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } }
I20260924 10:40:05.501279 8231 leader_election.cc:302] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee68668fd32841e2a0495725f909eee7; no voters:
I20260924 10:40:05.501631 8231 leader_election.cc:288] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:40:05.501689 8236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:05.501895 8236 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [term 1 LEADER]: Becoming Leader. State: Replica: ee68668fd32841e2a0495725f909eee7, State: Running, Role: LEADER
I20260924 10:40:05.502179 8236 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } }
I20260924 10:40:05.502905 8231 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:40:05.503885 8238 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee68668fd32841e2a0495725f909eee7. Latest consensus state: current_term: 1 leader_uuid: "ee68668fd32841e2a0495725f909eee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } }
I20260924 10:40:05.504024 8238 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:05.505123 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:05.491035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.2.8.62:38611)
0924 10:40:05.491519 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:05.491535 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:05.491580 (+ 45us) server_negotiation.cc:407] Connection header received
0924 10:40:05.492365 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:05.492386 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:05.492756 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:05.493163 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:05.493959 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:05.495031 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:05.495878 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:05.496371 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:05.496553 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:05.499696 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:05.499704 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:05.499722 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:05.499754 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:05.501643 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:05.502193 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:05.502198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:05.502204 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:05.502347 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:05.502646 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:05.502651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:05.502653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:05.503123 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:05.503343 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:05.503675 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 10:40:05.504026 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":144,"threads_started":1}
I20260924 10:40:05.505416 8237 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee68668fd32841e2a0495725f909eee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } }
I20260924 10:40:05.505507 8237 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: This master's current role is: LEADER
W20260924 10:40:05.507993 8251 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:40:05.508068 8251 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:40:05.508139 8245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:40:05.508956 8245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:40:05.513932 8245 catalog_manager.cc:1347] Generated new cluster ID: 11926dce6cf44ca1acfbebd864657b29
I20260924 10:40:05.513994 8245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:40:05.539543 8245 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:40:05.540593 8245 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:40:05.547189 8245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7: Generated new TSK 0
I20260924 10:40:05.547897 8245 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.1.keytab.
Entry for principal oryx/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.1.keytab.
I20260924 10:40:05.584527 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:38611
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.1.keytab
--principal=oryx/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:05.697564 8257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:05.697820 8257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:05.697862 8257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:05.701972 8257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:05.702029 8257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:05.702126 8257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:40:05.707551 8257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.1.keytab
--principal=oryx/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:38611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:05.708568 8257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:05.709573 8257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:05.718823 8263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:05.718825 8262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:05.719106 8265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:05.719085 8257 server_base.cc:1034] running on GCE node
I20260924 10:40:05.719583 8257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:05.720306 8257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:05.721488 8257 hybrid_clock.cc:648] HybridClock initialized: now 1790246405721462 us; error 48 us; skew 500 ppm
Sep 24 10:40:05 dist-test-slave-j38h krb5kdc[8150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246405, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:05.724218 8257 init.cc:377] Logged in from keytab as oryx/127.2.8.1@KRBTEST.COM (short username oryx)
I20260924 10:40:05.725147 8257 webserver.cc:460] Webserver started at http://127.2.8.1:34681/ using document root <none> and password file <none>
I20260924 10:40:05.725723 8257 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:05.725782 8257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:05.726008 8257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:05.727949 8257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "0bd5a2a65f554fdc95fe660cf67364ff"
format_stamp: "Formatted at 2026-09-24 10:40:05 on dist-test-slave-j38h"
server_key: "1af27d0e914c671794cfdbabb941e70a"
server_key_iv: "0d5d2caad75459c90799af8fd2b075f4"
server_key_version: "encryptionkey@0"
I20260924 10:40:05.728492 8257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "0bd5a2a65f554fdc95fe660cf67364ff"
format_stamp: "Formatted at 2026-09-24 10:40:05 on dist-test-slave-j38h"
server_key: "1af27d0e914c671794cfdbabb941e70a"
server_key_iv: "0d5d2caad75459c90799af8fd2b075f4"
server_key_version: "encryptionkey@0"
I20260924 10:40:05.732005 8257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 10:40:05.734453 8272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:05.735606 8257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:40:05.735714 8257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "0bd5a2a65f554fdc95fe660cf67364ff"
format_stamp: "Formatted at 2026-09-24 10:40:05 on dist-test-slave-j38h"
server_key: "1af27d0e914c671794cfdbabb941e70a"
server_key_iv: "0d5d2caad75459c90799af8fd2b075f4"
server_key_version: "encryptionkey@0"
I20260924 10:40:05.735812 8257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:05.746726 8257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:05.749925 8257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:05.750129 8257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:05.750701 8257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:05.751508 8257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:05.751551 8257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:05.751616 8257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:05.751646 8257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:05.760941 8257 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:42063
I20260924 10:40:05.761014 8385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:42063 every 8 connection(s)
I20260924 10:40:05.761860 8257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:40:05.770857 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 8257
I20260924 10:40:05.771006 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
Sep 24 10:40:05 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246405, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM
I20260924 10:40:05.771355 2080 external_mini_cluster.cc:1428] Setting key 30d85724bb664d3dbee5f181936bcd20
I20260924 10:40:05.775552 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:05.763714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:39749 (local address 127.2.8.62:38611)
0924 10:40:05.763848 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:05.763852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:05.764615 (+ 763us) server_negotiation.cc:407] Connection header received
0924 10:40:05.765969 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:05.765972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:05.766030 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:05.766129 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:05.767259 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:05.767787 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:05.768948 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:05.769249 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:05.769359 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:05.771803 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:05.771807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:05.771810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:05.771839 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:05.773350 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:05.773953 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:05.773955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:05.773956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:05.774026 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:05.774394 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:05.774396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:05.774398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:05.774653 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:05.774759 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:05.775211 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 10:40:05.775344 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:40:05.776233 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:05.764031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.1:39749)
0924 10:40:05.764465 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:05.764506 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:05.765742 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:05.766300 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:05.766306 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:05.766650 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:05.767106 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:05.767118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:05.767924 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:05.767928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:05.768807 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:05.768815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:05.769345 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:05.769396 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:05.769566 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:05.770101 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:05.770117 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:05.771661 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:05.773467 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:05.773475 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:05.773491 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:05.773846 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:05.774115 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:05.774121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:05.774126 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:05.774305 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:05.774761 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:05.774766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:05.774986 (+ 220us) client_negotiation.cc:769] Sending connection context
0924 10:40:05.775137 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 10:40:05.775366 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":116,"threads_started":1}
I20260924 10:40:05.777499 8386 heartbeater.cc:347] Connected to a master server at 127.2.8.62:38611
I20260924 10:40:05.777750 8386 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:05.778332 8386 heartbeater.cc:511] Master 127.2.8.62:38611 requested a full tablet report, sending...
I20260924 10:40:05.779964 8195 ts_manager.cc:194] Registered new tserver with Master: 0bd5a2a65f554fdc95fe660cf67364ff (127.2.8.1:42063)
WARNING: no policy specified for oryx/127.2.8.2@KRBTEST.COM; defaulting to no policy
I20260924 10:40:05.781181 8195 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:39749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.2.keytab.
Entry for principal oryx/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.2.keytab.
I20260924 10:40:05.799705 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:38611
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.2.keytab
--principal=oryx/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:05.907321 8391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:05.907593 8391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:05.907634 8391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:05.912119 8391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:05.912175 8391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:05.912233 8391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:40:05.917439 8391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.2.keytab
--principal=oryx/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:38611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:05.918375 8391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:05.919442 8391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:05.928777 8396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:05.928871 8391 server_base.cc:1034] running on GCE node
W20260924 10:40:05.928772 8399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:05.928838 8397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:05.929450 8391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:05.930064 8391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:05.931247 8391 hybrid_clock.cc:648] HybridClock initialized: now 1790246405931232 us; error 28 us; skew 500 ppm
Sep 24 10:40:05 dist-test-slave-j38h krb5kdc[8150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246405, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:05.934047 8391 init.cc:377] Logged in from keytab as oryx/127.2.8.2@KRBTEST.COM (short username oryx)
I20260924 10:40:05.934856 8391 webserver.cc:460] Webserver started at http://127.2.8.2:40375/ using document root <none> and password file <none>
I20260924 10:40:05.935400 8391 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:05.935489 8391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:05.935714 8391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:05.937752 8391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "6b40ad03a8884554a439d5ac6838d033"
format_stamp: "Formatted at 2026-09-24 10:40:05 on dist-test-slave-j38h"
server_key: "9d1a96dc669b237b1c415e7ed030d529"
server_key_iv: "44710e2487f18837b0fda01da1d2040c"
server_key_version: "encryptionkey@0"
I20260924 10:40:05.938392 8391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "6b40ad03a8884554a439d5ac6838d033"
format_stamp: "Formatted at 2026-09-24 10:40:05 on dist-test-slave-j38h"
server_key: "9d1a96dc669b237b1c415e7ed030d529"
server_key_iv: "44710e2487f18837b0fda01da1d2040c"
server_key_version: "encryptionkey@0"
I20260924 10:40:05.942361 8391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:40:05.944572 8406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:05.945773 8391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:40:05.945863 8391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "6b40ad03a8884554a439d5ac6838d033"
format_stamp: "Formatted at 2026-09-24 10:40:05 on dist-test-slave-j38h"
server_key: "9d1a96dc669b237b1c415e7ed030d529"
server_key_iv: "44710e2487f18837b0fda01da1d2040c"
server_key_version: "encryptionkey@0"
I20260924 10:40:05.945955 8391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:05.988126 8391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:05.991293 8391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:05.991509 8391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:05.992049 8391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:05.992826 8391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:05.992872 8391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:05.992911 8391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:05.992926 8391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:06.001796 8391 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:35103
I20260924 10:40:06.001824 8519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:35103 every 8 connection(s)
I20260924 10:40:06.002655 8391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:40:06.006127 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 8391
I20260924 10:40:06.006218 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:40:06.006496 2080 external_mini_cluster.cc:1428] Setting key b730bcf64cb10951366b7454fa1aff03
Sep 24 10:40:06 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246405, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.2.8.3@KRBTEST.COM; defaulting to no policy
I20260924 10:40:06.017292 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.004738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:43377 (local address 127.2.8.62:38611)
0924 10:40:06.004970 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.004994 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.005844 (+ 850us) server_negotiation.cc:407] Connection header received
0924 10:40:06.007441 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.007445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.007496 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.007573 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.008698 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.009281 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.010412 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.010625 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.010735 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.013534 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.013537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.013540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.013571 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.015078 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.015604 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.015610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.015612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.015690 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.016033 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.016036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.016037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.016313 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.016426 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.016912 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.017085 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.8.3@KRBTEST.COM" created.
I20260924 10:40:06.018219 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.005242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.2:43377)
0924 10:40:06.005676 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.005718 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.007177 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.007735 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.007742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.008083 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:06.008554 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.008567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.009401 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.009405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.010281 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.010289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.010719 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.010773 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.010952 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.011566 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:06.011582 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:06.013365 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:06.015210 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.015216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.015231 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.015503 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.015789 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.015792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.015794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.015929 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.016423 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:06.016429 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:06.016674 (+ 245us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.016872 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.017225 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
I20260924 10:40:06.020403 8520 heartbeater.cc:347] Connected to a master server at 127.2.8.62:38611
I20260924 10:40:06.020645 8520 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:06.021209 8520 heartbeater.cc:511] Master 127.2.8.62:38611 requested a full tablet report, sending...
I20260924 10:40:06.022348 8195 ts_manager.cc:194] Registered new tserver with Master: 6b40ad03a8884554a439d5ac6838d033 (127.2.8.2:35103)
I20260924 10:40:06.023242 8195 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:43377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.3.keytab.
Entry for principal oryx/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.3.keytab.
I20260924 10:40:06.034183 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:38611
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.3.keytab
--principal=oryx/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:06.143947 8525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:06.144196 8525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:06.144237 8525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:06.148303 8525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:06.148362 8525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:06.148420 8525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:40:06.153774 8525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.3.keytab
--principal=oryx/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:38611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:06.154882 8525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:06.155896 8525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:06.165365 8530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:06.165427 8531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:06.165380 8533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:06.165572 8525 server_base.cc:1034] running on GCE node
I20260924 10:40:06.166013 8525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:06.166656 8525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:06.167842 8525 hybrid_clock.cc:648] HybridClock initialized: now 1790246406167806 us; error 29 us; skew 500 ppm
Sep 24 10:40:06 dist-test-slave-j38h krb5kdc[8150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246406, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:06.170697 8525 init.cc:377] Logged in from keytab as oryx/127.2.8.3@KRBTEST.COM (short username oryx)
I20260924 10:40:06.171578 8525 webserver.cc:460] Webserver started at http://127.2.8.3:35585/ using document root <none> and password file <none>
I20260924 10:40:06.172154 8525 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:06.172240 8525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:06.172468 8525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:06.174515 8525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a"
format_stamp: "Formatted at 2026-09-24 10:40:06 on dist-test-slave-j38h"
server_key: "7e5fe8e803715548b2a5f63f87cc87bd"
server_key_iv: "8aacb5352099fd06385f6070e8c7cc96"
server_key_version: "encryptionkey@0"
I20260924 10:40:06.175064 8525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a"
format_stamp: "Formatted at 2026-09-24 10:40:06 on dist-test-slave-j38h"
server_key: "7e5fe8e803715548b2a5f63f87cc87bd"
server_key_iv: "8aacb5352099fd06385f6070e8c7cc96"
server_key_version: "encryptionkey@0"
I20260924 10:40:06.178781 8525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 10:40:06.181191 8540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:06.182377 8525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:40:06.182544 8525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a"
format_stamp: "Formatted at 2026-09-24 10:40:06 on dist-test-slave-j38h"
server_key: "7e5fe8e803715548b2a5f63f87cc87bd"
server_key_iv: "8aacb5352099fd06385f6070e8c7cc96"
server_key_version: "encryptionkey@0"
I20260924 10:40:06.182653 8525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:06.202510 8525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:06.205655 8525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:06.205855 8525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:06.206449 8525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:06.207273 8525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:06.207342 8525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:06.207418 8525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:06.207463 8525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:06.216329 8525 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:43511
I20260924 10:40:06.216347 8653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:43511 every 8 connection(s)
I20260924 10:40:06.217284 8525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:40:06.220512 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 8525
I20260924 10:40:06.220599 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:40:06.220887 2080 external_mini_cluster.cc:1428] Setting key 5475c2c2295b7f62988fdc15ade6ad97
Sep 24 10:40:06 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246406, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM
I20260924 10:40:06.231120 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.219004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:38771 (local address 127.2.8.62:38611)
0924 10:40:06.219148 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.219151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.220034 (+ 883us) server_negotiation.cc:407] Connection header received
0924 10:40:06.221614 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.221617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.221658 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.221761 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.223288 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.223850 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.225000 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.225287 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.225394 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.227646 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.227649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.227651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.227675 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.229261 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.229758 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.229760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.229762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.229857 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.230150 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.230152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.230153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.230342 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.230423 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.230906 (+ 483us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.231006 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:40:06.231882 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.219370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.3:38771)
0924 10:40:06.219853 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.219893 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.221394 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.221977 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.221984 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.222483 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:06.223137 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.223156 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.223970 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.223972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.224831 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.224840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.225383 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.225424 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.225571 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.226076 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:06.226091 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:06.227481 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:06.229371 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.229377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.229392 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.229666 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.229943 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.229945 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.229946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.230071 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.230433 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:06.230449 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:06.230665 (+ 216us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.230820 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.231062 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":130,"threads_started":1}
I20260924 10:40:06.233042 8654 heartbeater.cc:347] Connected to a master server at 127.2.8.62:38611
I20260924 10:40:06.233302 8654 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:06.233786 8654 heartbeater.cc:511] Master 127.2.8.62:38611 requested a full tablet report, sending...
I20260924 10:40:06.234802 8194 ts_manager.cc:194] Registered new tserver with Master: ad2b0cf23c7c40fd8509b8438ea59d1a (127.2.8.3:43511)
I20260924 10:40:06.235394 8194 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:38771
I20260924 10:40:06.246343 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.8.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.8.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.8.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.61.keytab.
Entry for principal oryx/127.2.8.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.61.keytab.
I20260924 10:40:06.272692 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.61:37391
--webserver_interface=127.2.8.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.61.keytab
--principal=oryx/127.2.8.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:38611,127.2.8.61:37391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:06.382754 8659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:06.382994 8659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:06.383033 8659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:06.387084 8659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:40:06.387135 8659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:06.387154 8659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:06.387194 8659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:40:06.387229 8659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:40:06.392259 8659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:38611,127.2.8.61:37391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.61.keytab
--principal=oryx/127.2.8.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.61:37391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.8.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
W20260924 10:40:06.392481 8659 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.8.62:38611,127.2.8.61:37391'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 10:40:06.393376 8659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:06.394368 8659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:06.402649 8665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:06.402765 8667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:06.402649 8664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:06.403002 8659 server_base.cc:1034] running on GCE node
I20260924 10:40:06.403577 8659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:06.404723 8659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:06.405916 8659 hybrid_clock.cc:648] HybridClock initialized: now 1790246406405872 us; error 50 us; skew 500 ppm
Sep 24 10:40:06 dist-test-slave-j38h krb5kdc[8150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246406, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:06.408651 8659 init.cc:377] Logged in from keytab as oryx/127.2.8.61@KRBTEST.COM (short username oryx)
I20260924 10:40:06.409612 8659 webserver.cc:460] Webserver started at http://127.2.8.61:40577/ using document root <none> and password file <none>
I20260924 10:40:06.410128 8659 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:06.410190 8659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:06.410363 8659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:06.412410 8659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/data/instance:
uuid: "3192885cd0a94303817ce469f56c9ed2"
format_stamp: "Formatted at 2026-09-24 10:40:06 on dist-test-slave-j38h"
server_key: "80b00c1cc098b8ea2d4d9ddde91918c5"
server_key_iv: "e86adec1001ad1cebfad2456e341684a"
server_key_version: "encryptionkey@0"
I20260924 10:40:06.412925 8659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/wal/instance:
uuid: "3192885cd0a94303817ce469f56c9ed2"
format_stamp: "Formatted at 2026-09-24 10:40:06 on dist-test-slave-j38h"
server_key: "80b00c1cc098b8ea2d4d9ddde91918c5"
server_key_iv: "e86adec1001ad1cebfad2456e341684a"
server_key_version: "encryptionkey@0"
I20260924 10:40:06.416656 8659 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:40:06.419149 8674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:06.420558 8659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:40:06.420682 8659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/wal
uuid: "3192885cd0a94303817ce469f56c9ed2"
format_stamp: "Formatted at 2026-09-24 10:40:06 on dist-test-slave-j38h"
server_key: "80b00c1cc098b8ea2d4d9ddde91918c5"
server_key_iv: "e86adec1001ad1cebfad2456e341684a"
server_key_version: "encryptionkey@0"
I20260924 10:40:06.420794 8659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:06.445626 8659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:06.448954 8659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:06.449218 8659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:06.456878 8659 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.61:37391
I20260924 10:40:06.457201 8726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.61:37391 every 8 connection(s)
I20260924 10:40:06.457916 8659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/data/info.pb
I20260924 10:40:06.458529 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 8659
I20260924 10:40:06.461110 8727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:06.464700 8727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:40:06 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.8.61@KRBTEST.COM
Sep 24 10:40:06 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246406, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.61@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM
I20260924 10:40:06.474351 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.460082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42222 (local address 127.2.8.61:37391)
0924 10:40:06.460538 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.460554 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.460591 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:40:06.461463 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.461483 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.461866 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.462220 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.463076 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.464158 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.465118 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.465780 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.465941 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.469058 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.469065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.469075 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.469102 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.471198 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.471701 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.471706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.471710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.471809 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.472138 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.472140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.472141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.472599 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.472847 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.473112 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.473375 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"spinlock_wait_cycles":5504,"thread_start_us":161,"threads_started":1}
I20260924 10:40:06.476193 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.466666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.2.8.62:38611)
0924 10:40:06.466827 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.466832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.467171 (+ 339us) server_negotiation.cc:407] Connection header received
0924 10:40:06.467317 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.467322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.467416 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.467492 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.468219 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.468820 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.469950 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.470166 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.470274 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.472698 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.472707 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.472709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.472734 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.474336 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.474850 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.474854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.474856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.474944 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.475223 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.475226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.475227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.475506 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.475604 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.475928 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.476037 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:40:06.476217 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.466617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.0.0.1:60890)
0924 10:40:06.467016 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.467075 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.467209 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.467565 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.467573 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.467968 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:06.468091 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.468096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.468948 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.468951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.469815 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.469825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.470271 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.470286 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.470414 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.470881 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:06.470892 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:06.472553 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:06.474475 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.474486 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.474495 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.474753 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.475043 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.475046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.475047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.475126 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.475610 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:06.475618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:06.475728 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.475867 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.476039 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":142,"threads_started":1}
I20260924 10:40:06.477378 8727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:40:06 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246406, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.61@KRBTEST.COM for oryx/127.2.8.61@KRBTEST.COM
I20260924 10:40:06.486083 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.477846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:37391 (local address 127.0.0.1:42224)
0924 10:40:06.477965 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.477980 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.478059 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.478252 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.478254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.478424 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:06.478510 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.478515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.479797 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.479799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.480368 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.480376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.480902 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.480915 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.481054 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.481512 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:06.481523 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:06.482893 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:06.484594 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.484597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.484598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.484781 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.485124 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.485127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.485128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.485214 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.485672 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:06.485674 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:06.485757 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.485856 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.485964 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 10:40:06.486207 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.477916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.2.8.61:37391)
0924 10:40:06.478019 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.478022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.478032 (+ 10us) server_negotiation.cc:407] Connection header received
0924 10:40:06.478128 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.478130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.478166 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.478220 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.478607 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.479686 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.480481 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.480783 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.480891 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.483013 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.483017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.483019 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.483042 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.484489 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.484870 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.484873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.484875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.484949 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.485370 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.485373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.485375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.485574 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.485662 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.485932 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.486038 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:40:06.489262 8727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: Bootstrap starting.
I20260924 10:40:06.491523 8727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:06.492264 8727 log.cc:824] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:06.494269 8727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: No bootstrap required, opened a new log
I20260924 10:40:06.497234 8727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } }
I20260924 10:40:06.497481 8727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:06.497570 8727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3192885cd0a94303817ce469f56c9ed2, State: Initialized, Role: FOLLOWER
I20260924 10:40:06.498016 8727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } }
I20260924 10:40:06.498478 8733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } } }
I20260924 10:40:06.498584 8733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:40:06.499185 8727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 10:40:06.504060 8744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:40:06.504143 8744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:40:06.505910 8659 master_runner.cc:428] Detected that this master 3192885cd0a94303817ce469f56c9ed2 is joining an existing cluster
I20260924 10:40:06.505990 8659 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.8.61:37391
I20260924 10:40:06.516264 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.508339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.2.8.61:37391)
0924 10:40:06.508428 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.508432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.508508 (+ 76us) server_negotiation.cc:407] Connection header received
0924 10:40:06.508650 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.508652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.508685 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.508779 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.509310 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.509898 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.510920 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.511135 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.511238 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.512551 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.512555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.512557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.512577 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.514188 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.514829 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.514832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.514833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.514901 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.515172 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.515175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.515177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.515338 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.515439 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.515701 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.515888 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 10:40:06.516680 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.508000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:37391 (local address 127.0.0.1:42238)
0924 10:40:06.508439 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.508453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.508564 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.508797 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.508800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.509053 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:06.509189 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.509197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.510006 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.510009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.510803 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.510810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.511243 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.511257 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.511399 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.511991 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:06.512004 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:06.512415 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:06.514331 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.514334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.514336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.514516 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.515001 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.515004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.515006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.515081 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.515427 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:06.515430 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:06.515515 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.515786 (+ 271us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.516520 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":617,"thread_start_us":87,"threads_started":1}
I20260924 10:40:06.517169 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.507966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.2.8.62:38611)
0924 10:40:06.508097 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.508101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.508854 (+ 753us) server_negotiation.cc:407] Connection header received
0924 10:40:06.509042 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.509045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.509094 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.509178 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.509692 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.510455 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.511802 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.512145 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.512251 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.513732 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.513736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.513740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.513769 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.515473 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.515906 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.515911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.515913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.515993 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.516262 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.516265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.516266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.516498 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.516581 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.516892 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.517033 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:40:06.517603 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.508074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.0.0.1:60904)
0924 10:40:06.508782 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.508794 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.508912 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.509216 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.509219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.509446 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:06.509570 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.509577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.510568 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.510573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.511623 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.511633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.512256 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.512271 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.512377 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.512837 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:06.512849 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:06.513366 (+ 517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:06.515572 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.515575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.515576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.515828 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.516088 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.516091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.516093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.516171 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.516620 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:06.516623 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:06.516704 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.517308 (+ 604us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.517432 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":80,"thread_start_us":91,"threads_started":1}
Sep 24 10:40:06 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246405, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.62@KRBTEST.COM for oryx/127.2.8.61@KRBTEST.COM
I20260924 10:40:06.531550 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.521737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:37391 (local address 127.0.0.1:42250)
0924 10:40:06.522076 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.522118 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.522334 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.522665 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.522672 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.523049 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:06.523152 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.523159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.523907 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.523911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.524642 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.524652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.525204 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.525221 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.525359 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.525930 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:06.525943 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:06.527543 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:06.529519 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.529529 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.529535 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.529792 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.530156 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.530161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.530165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.530279 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.530870 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:06.530877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:06.531049 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.531201 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.531417 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":79,"threads_started":1}
I20260924 10:40:06.532095 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.521833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42250 (local address 127.2.8.61:37391)
0924 10:40:06.521949 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.521954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.522201 (+ 247us) server_negotiation.cc:407] Connection header received
0924 10:40:06.522476 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.522480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.522532 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.522633 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.523288 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.523785 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.524784 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.525095 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.525218 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.527698 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.527701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.527703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.527752 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.529381 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.529918 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.529925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.529928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.530027 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.530412 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.530417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.530419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.530723 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.531745 (+ 1022us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.531827 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.531956 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:40:06.533371 8194 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.8.61:37391: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3192885cd0a94303817ce469f56c9ed2"
member_type: NON_VOTER
last_known_addr {
host: "127.2.8.61"
port: 37391
}
attrs {
promote: true
}
}
dest_uuid: "ee68668fd32841e2a0495725f909eee7"
cas_config_opid_index: -1
I20260924 10:40:06.534070 8194 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: NON_VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: true } }
I20260924 10:40:06.535768 8754 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3192885cd0a94303817ce469f56c9ed2 (127.2.8.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: NON_VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: true } } }
I20260924 10:40:06.536407 8701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:06.537645 8754 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.8.61:37391
I20260924 10:40:06.537782 8701 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 1 FOLLOWER]: Refusing update from remote peer ee68668fd32841e2a0495725f909eee7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 10:40:06.538044 8659 master.cc:537] Master@127.2.8.61:37391 shutting down...
I20260924 10:40:06.538762 8754 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: NON_VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:06.539867 8756 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ee68668fd32841e2a0495725f909eee7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: NON_VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: true } } }
I20260924 10:40:06.539973 8756 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: This master's current role is: LEADER
W20260924 10:40:06.540591 8179 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
WARNING: no policy specified for oryx/127.2.8.60@KRBTEST.COM; defaulting to no policy
W20260924 10:40:06.542375 8179 consensus_peers.cc:487] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 -> Peer 3192885cd0a94303817ce469f56c9ed2 (127.2.8.61:37391): Couldn't send request to peer 3192885cd0a94303817ce469f56c9ed2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 10:40:06.543329 8659 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:40:06.543594 8659 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:06.543694 8659 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.8.60@KRBTEST.COM" created.
I20260924 10:40:06.548457 8659 master.cc:552] Master@127.2.8.61:37391 shutdown complete.
I20260924 10:40:06.548519 8659 master_runner.cc:305] Clearing existing system tablet
I20260924 10:40:06.548779 8659 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:06.548846 8659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:06.552577 8659 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:40:06.553623 8761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:06.553891 8659 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:06.554041 8659 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 10:40:06.554075 8659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/wal
uuid: "3192885cd0a94303817ce469f56c9ed2"
format_stamp: "Formatted at 2026-09-24 10:40:06 on dist-test-slave-j38h"
server_key: "80b00c1cc098b8ea2d4d9ddde91918c5"
server_key_iv: "e86adec1001ad1cebfad2456e341684a"
server_key_version: "encryptionkey@0"
I20260924 10:40:06.556015 8659 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:06.558907 8659 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 10:40:06.558979 8659 log.cc:1196] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 10:40:06.559257 8659 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: Deleting consensus metadata
I20260924 10:40:06.559475 8659 master_runner.cc:315] Copying system tablet from 127.2.8.62:38611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.8.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.60.keytab.
Entry for principal oryx/127.2.8.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.60.keytab.
I20260924 10:40:06.560794 8659 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.8.62:38611
I20260924 10:40:06.560832 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.60:33075
--webserver_interface=127.2.8.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.60.keytab
--principal=oryx/127.2.8.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:38611,127.2.8.61:37391,127.2.8.60:33075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:40:06.569936 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.561655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.0.0.1:60906)
0924 10:40:06.561870 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.561884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.562067 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.562509 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.562514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.562743 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:06.563192 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.563202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.564401 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.564404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.565193 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.565201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.565646 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.565666 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.565749 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.566292 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:06.566312 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:06.566630 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:06.568368 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.568373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.568376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.568575 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.568857 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.568862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.568864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.568950 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.569359 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:06.569362 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:06.569463 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.569571 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.569722 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
I20260924 10:40:06.569926 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.561558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60906 (local address 127.2.8.62:38611)
0924 10:40:06.561694 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.561700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.561968 (+ 268us) server_negotiation.cc:407] Connection header received
0924 10:40:06.562194 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.562199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.562261 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.562335 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.563354 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.564207 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.565309 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.565549 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.565658 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.566768 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.566772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.566774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.566801 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.568250 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.568675 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.568679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.568681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.568760 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.569079 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.569083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.569084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.569258 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.569368 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.569655 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.569764 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:40:06.570874 8215 tablet_copy_service.cc:140] P ee68668fd32841e2a0495725f909eee7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3192885cd0a94303817ce469f56c9ed2 ({username='oryx', principal='oryx/127.2.8.61@KRBTEST.COM'} at 127.0.0.1:60906)
I20260924 10:40:06.571053 8215 tablet_copy_service.cc:161] P ee68668fd32841e2a0495725f909eee7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3192885cd0a94303817ce469f56c9ed2 at {username='oryx', principal='oryx/127.2.8.61@KRBTEST.COM'} at 127.0.0.1:60906: session id = 3192885cd0a94303817ce469f56c9ed2-00000000000000000000000000000000
I20260924 10:40:06.572528 8215 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 10:40:06.574060 8659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 10:40:06.575947 8659 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: tablet copy: Starting download of 0 data blocks...
I20260924 10:40:06.576138 8659 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: tablet copy: Starting download of 1 WAL segments...
I20260924 10:40:06.577813 8659 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 10:40:06.579392 8215 tablet_copy_service.cc:342] P ee68668fd32841e2a0495725f909eee7: Request end of tablet copy session 3192885cd0a94303817ce469f56c9ed2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.8.61@KRBTEST.COM'} at 127.0.0.1:60906
I20260924 10:40:06.579509 8215 tablet_copy_service.cc:433] P ee68668fd32841e2a0495725f909eee7: ending tablet copy session 3192885cd0a94303817ce469f56c9ed2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3192885cd0a94303817ce469f56c9ed2
I20260924 10:40:06.586531 8659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:06.588133 8769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:06.588191 8772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:06.588551 8770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:06.588917 8659 server_base.cc:1034] running on GCE node
I20260924 10:40:06.589119 8659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:06.589489 8659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:06.590651 8659 hybrid_clock.cc:648] HybridClock initialized: now 1790246406590640 us; error 28 us; skew 500 ppm
Sep 24 10:40:06 dist-test-slave-j38h krb5kdc[8150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246406, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:06.592161 8659 init.cc:377] Logged in from keytab as oryx/127.2.8.61@KRBTEST.COM (short username oryx)
I20260924 10:40:06.592617 8659 webserver.cc:460] Webserver started at http://127.2.8.61:33809/ using document root <none> and password file <none>
I20260924 10:40:06.592803 8659 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:06.592856 8659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:06.594002 8659 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:40:06.594901 8779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:06.595139 8659 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:40:06.595228 8659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/wal
uuid: "3192885cd0a94303817ce469f56c9ed2"
format_stamp: "Formatted at 2026-09-24 10:40:06 on dist-test-slave-j38h"
server_key: "80b00c1cc098b8ea2d4d9ddde91918c5"
server_key_iv: "e86adec1001ad1cebfad2456e341684a"
server_key_version: "encryptionkey@0"
I20260924 10:40:06.595311 8659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:06.618268 8659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:06.620406 8659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:06.625267 8659 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.61:37391
I20260924 10:40:06.625276 8831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.61:37391 every 8 connection(s)
I20260924 10:40:06.625821 8659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-1/data/info.pb
I20260924 10:40:06.627228 8832 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:40:06.628032 8832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: Bootstrap starting.
I20260924 10:40:06.635962 8832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:40:06.636273 8832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: Bootstrap complete.
I20260924 10:40:06.636768 8832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: NON_VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: true } }
I20260924 10:40:06.636900 8832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3192885cd0a94303817ce469f56c9ed2, State: Initialized, Role: LEARNER
I20260924 10:40:06.637030 8832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: NON_VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: true } }
I20260924 10:40:06.637367 8835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: NON_VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: true } } }
I20260924 10:40:06.637521 8835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: This master's current role is: LEARNER
I20260924 10:40:06.637534 8832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:40:06.640398 8846 catalog_manager.cc:1259] Loaded cluster ID: 11926dce6cf44ca1acfbebd864657b29
I20260924 10:40:06.640532 8846 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: loading cluster ID for follower catalog manager: success
I20260924 10:40:06.642333 8846 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: acquiring CA information for follower catalog manager: success
I20260924 10:40:06.643378 8846 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 10:40:06.714128 8768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:06.714394 8768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:06.714434 8768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:06.718628 8768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:40:06.718688 8768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:06.718707 8768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:06.718725 8768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:40:06.718780 8768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:40:06.724373 8768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:38611,127.2.8.61:37391,127.2.8.60:33075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.60.keytab
--principal=oryx/127.2.8.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.60:33075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.8.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:06.725602 8768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:06.726670 8768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:06.735289 8852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:06.735304 8854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:06.735461 8851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:06.735728 8768 server_base.cc:1034] running on GCE node
I20260924 10:40:06.736299 8768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:06.737560 8768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:06.738739 8768 hybrid_clock.cc:648] HybridClock initialized: now 1790246406738724 us; error 35 us; skew 500 ppm
Sep 24 10:40:06 dist-test-slave-j38h krb5kdc[8150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246406, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:06.741751 8768 init.cc:377] Logged in from keytab as oryx/127.2.8.60@KRBTEST.COM (short username oryx)
I20260924 10:40:06.742611 8768 webserver.cc:460] Webserver started at http://127.2.8.60:38667/ using document root <none> and password file <none>
I20260924 10:40:06.743165 8768 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:06.743255 8768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:06.743484 8768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:06.745565 8768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/data/instance:
uuid: "448a89e65a7c40c3b72df3b2ba71e68f"
format_stamp: "Formatted at 2026-09-24 10:40:06 on dist-test-slave-j38h"
server_key: "809ffb3635f84b2798bf892607d53f28"
server_key_iv: "c8dfac7c6bcfe982e9b9e240b5bd27e3"
server_key_version: "encryptionkey@0"
I20260924 10:40:06.746119 8768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/wal/instance:
uuid: "448a89e65a7c40c3b72df3b2ba71e68f"
format_stamp: "Formatted at 2026-09-24 10:40:06 on dist-test-slave-j38h"
server_key: "809ffb3635f84b2798bf892607d53f28"
server_key_iv: "c8dfac7c6bcfe982e9b9e240b5bd27e3"
server_key_version: "encryptionkey@0"
I20260924 10:40:06.749948 8768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 10:40:06.752357 8861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:06.753726 8768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:40:06.753881 8768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/wal
uuid: "448a89e65a7c40c3b72df3b2ba71e68f"
format_stamp: "Formatted at 2026-09-24 10:40:06 on dist-test-slave-j38h"
server_key: "809ffb3635f84b2798bf892607d53f28"
server_key_iv: "c8dfac7c6bcfe982e9b9e240b5bd27e3"
server_key_version: "encryptionkey@0"
I20260924 10:40:06.754004 8768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:06.783577 8386 heartbeater.cc:503] Master 127.2.8.62:38611 was elected leader, sending a full tablet report...
I20260924 10:40:06.789674 8768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:06.792891 8768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:06.793146 8768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:06.800285 8768 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.60:33075
I20260924 10:40:06.800585 8913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.60:33075 every 8 connection(s)
I20260924 10:40:06.801303 8768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/data/info.pb
I20260924 10:40:06.801627 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 8768
I20260924 10:40:06.804584 8914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:06.809108 8914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:40:06 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.8.60@KRBTEST.COM
Sep 24 10:40:06 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246406, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.60@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM
I20260924 10:40:06.817981 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.803586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.2.8.60:33075)
0924 10:40:06.804044 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.804063 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.804103 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:40:06.805017 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.805039 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.805437 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.805781 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.806498 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.807974 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.808826 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.809529 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.809674 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.812763 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.812769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.812778 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.812803 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.814747 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.815206 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.815210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.815215 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.815373 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.815650 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.815653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.815655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.816182 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.816394 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.816662 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.816932 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":161,"threads_started":1}
I20260924 10:40:06.820909 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.811034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.0.0.1:60914)
0924 10:40:06.811448 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.811516 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.811643 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.811959 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.811965 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.812439 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:06.812610 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.812617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.813609 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.813615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.814683 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.814695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.815270 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.815291 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.815406 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.815869 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:06.815882 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:06.817393 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:06.819184 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.819195 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.819201 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.819419 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.819710 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.819713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.819715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.819796 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.820248 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:06.820255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:06.820367 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.820513 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.820687 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":169,"threads_started":1}
I20260924 10:40:06.820998 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.811152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60914 (local address 127.2.8.62:38611)
0924 10:40:06.811281 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.811287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.811608 (+ 321us) server_negotiation.cc:407] Connection header received
0924 10:40:06.811743 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.811747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.811800 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.811895 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.812776 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.813497 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.814836 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.815178 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.815269 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.817535 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.817538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.817541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.817567 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.819051 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.819522 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.819527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.819529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.819611 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.819902 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.819905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.819907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.820132 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.820247 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.820601 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.820818 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:40:06.821910 8914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:40:06 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246406, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.60@KRBTEST.COM for oryx/127.2.8.61@KRBTEST.COM
I20260924 10:40:06.830140 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.822451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:37391 (local address 127.0.0.1:42262)
0924 10:40:06.822620 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.822635 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.822724 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.823144 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.823147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.823267 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:06.823350 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.823355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.824215 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.824217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.824731 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.824738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.825267 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.825278 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.825343 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.825709 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:06.825719 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:06.827305 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:06.828953 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.828955 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.828956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.829177 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.829403 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.829405 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.829406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.829489 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.829791 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:06.829793 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:06.829859 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.829933 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.830026 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 10:40:06.830298 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.822651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42262 (local address 127.2.8.61:37391)
0924 10:40:06.822934 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.822938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.822950 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:40:06.822984 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.822986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.823021 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.823095 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.823453 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.824124 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.824848 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.825160 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.825258 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.827430 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.827433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.827436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.827457 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.828855 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.829255 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.829257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.829259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.829327 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.829561 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.829563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.829564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.829719 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.829790 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.829996 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.830111 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":107,"threads_started":1}
I20260924 10:40:06.831018 8914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:40:06 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246406, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.60@KRBTEST.COM for oryx/127.2.8.60@KRBTEST.COM
I20260924 10:40:06.838913 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.831471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:33075 (local address 127.0.0.1:49098)
0924 10:40:06.831571 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.831582 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.831640 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.831818 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.831820 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.831937 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:06.832003 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.832007 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.832833 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.832836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.833403 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.833411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.833841 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.833854 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.833909 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.834365 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:06.834377 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:06.835593 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:06.837373 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.837376 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.837377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.837566 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.837890 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.837893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.837894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.837976 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.838329 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:06.838331 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:06.838403 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.838546 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.838658 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 10:40:06.838964 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.831502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49098 (local address 127.2.8.60:33075)
0924 10:40:06.831628 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.831631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.831642 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:40:06.831704 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.831706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.831737 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.831796 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.832154 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.832728 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.833534 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.833750 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.833838 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.835716 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.835719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.835720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.835740 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.837275 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.837684 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.837691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.837692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.837765 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.838068 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.838071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.838072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.838230 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.838305 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.838634 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.838745 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:40:06.841904 8914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: Bootstrap starting.
I20260924 10:40:06.844029 8914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:06.844674 8914 log.cc:824] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:06.846448 8914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: No bootstrap required, opened a new log
I20260924 10:40:06.849454 8914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } }
I20260924 10:40:06.849642 8914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:06.849676 8914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448a89e65a7c40c3b72df3b2ba71e68f, State: Initialized, Role: FOLLOWER
I20260924 10:40:06.850090 8914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } }
I20260924 10:40:06.850540 8920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } } }
I20260924 10:40:06.850677 8920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:40:06.851325 8914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [sys.catalog]: configured and running, proceeding with master startup.
W20260924 10:40:06.856278 8931 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:40:06.856354 8931 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:40:06.861636 8768 master_runner.cc:428] Detected that this master 448a89e65a7c40c3b72df3b2ba71e68f is joining an existing cluster
I20260924 10:40:06.861703 8768 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.8.60:33075
I20260924 10:40:06.871385 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 8257
I20260924 10:40:06.879245 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:42063
--local_ip_for_outbound_sockets=127.2.8.1
--tserver_master_addrs=127.2.8.62:38611
--webserver_port=34681
--webserver_interface=127.2.8.1
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.1.keytab
--principal=oryx/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.8.62:38611,127.2.8.61:37391
--tserver_master_addrs=127.2.8.62:38611,127.2.8.61:37391,127.2.8.60:33075 with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:40:06.887593 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.865858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.2.8.62:38611)
0924 10:40:06.865964 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.865968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.869004 (+ 3036us) server_negotiation.cc:407] Connection header received
0924 10:40:06.869191 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.869194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.869241 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.869291 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.870006 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.871322 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.872320 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.872647 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.872717 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.881864 (+ 9147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.881867 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.881869 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.881895 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.885901 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.886346 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.886350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.886352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.886439 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.886769 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.886772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.886774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.886962 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.887068 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.887322 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.887444 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 10:40:06.887609 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.865999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.0.0.1:60928)
0924 10:40:06.868892 (+ 2893us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.868907 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.869080 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.869441 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.869444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.869685 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:06.869825 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.869832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.871454 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.871457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.872178 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.872187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.872871 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.872886 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.873002 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.881094 (+ 8092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:06.881111 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:06.881695 (+ 584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:06.886021 (+ 4326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.886024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.886026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.886241 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.886532 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.886536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.886538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.886673 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.887056 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:06.887059 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:06.887161 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.887262 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.887410 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2779,"thread_start_us":79,"threads_started":1}
Sep 24 10:40:06 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246405, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.62@KRBTEST.COM for oryx/127.2.8.60@KRBTEST.COM
I20260924 10:40:06.906026 8939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.869755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:37391 (local address 127.0.0.1:42274)
0924 10:40:06.873359 (+ 3604us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.873372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.873499 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.873843 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.873846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.897147 (+ 23301us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:06.897271 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.897279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.898778 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.898781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.899720 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.899730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.900455 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.900469 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.900558 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.901073 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:06.901091 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:06.901833 (+ 742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:06.904321 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.904324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.904327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.904574 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.904861 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.904866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.904871 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.904943 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.905442 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:06.905445 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:06.905541 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.905628 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.905793 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3526,"mutex_wait_us":2,"spinlock_wait_cycles":9984,"thread_start_us":82,"threads_started":1}
I20260924 10:40:06.906544 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.865297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:33075 (local address 127.0.0.1:49108)
0924 10:40:06.868775 (+ 3478us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.868791 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.868933 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.869274 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.869276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.869577 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:06.870088 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.870098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.871054 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.871059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.873444 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.873458 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.874197 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.874215 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.874356 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.898529 (+ 24173us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:06.898544 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:06.899042 (+ 498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:06.904438 (+ 5396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.904441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.904443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.904715 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.905026 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.905029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.905031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.905105 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.905477 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:06.905480 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:06.905566 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.905645 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.906336 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3378,"mutex_wait_us":578,"thread_start_us":100,"threads_started":1}
I20260924 10:40:06.907229 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.869859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42274 (local address 127.2.8.61:37391)
0924 10:40:06.869986 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.869990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.873440 (+ 3450us) server_negotiation.cc:407] Connection header received
0924 10:40:06.873607 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.873610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.873659 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.873706 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.897548 (+ 23842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.898354 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.899863 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.900248 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.900352 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.901992 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.901996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.902001 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.902036 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.904156 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.904679 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.904684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.904687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.904766 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.905077 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.905079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.905081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.905301 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.905389 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.906742 (+ 1353us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.906879 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:40:06.908617 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.865329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.2.8.60:33075)
0924 10:40:06.865449 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.865453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.868857 (+ 3404us) server_negotiation.cc:407] Connection header received
0924 10:40:06.869059 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.869062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.869101 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.869144 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.870214 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.870920 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.873571 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.873982 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.874069 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.901310 (+ 27241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.901316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.901318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.901350 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.904337 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.904825 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.904829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.904832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.904913 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.905193 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.905196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.905197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.905372 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.905449 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.905724 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.908447 (+ 2723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2591,"server-negotiator.queue_time_us":39}
I20260924 10:40:06.909953 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.891490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:33075 (local address 127.0.0.1:49114)
0924 10:40:06.891661 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.891677 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.891752 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.892040 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.892043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.892219 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:06.892295 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.892300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.893290 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.893294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.893829 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.893836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.894343 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.894358 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.894453 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.894880 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:06.894894 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:06.897878 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:06.907952 (+ 10074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.907955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.907957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.908227 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.908554 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.908558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.908560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.908654 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.909432 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:06.909435 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:06.909563 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.909684 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.909835 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 10:40:06.910511 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.891535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.2.8.60:33075)
0924 10:40:06.891780 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.891791 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.891805 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:40:06.891868 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.891870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.891933 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.892026 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.892405 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.892926 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.893947 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.894244 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.894355 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.898041 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.898047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.898054 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.898084 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.907798 (+ 9714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.908343 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.908346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.908351 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.908446 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.909066 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.909072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.909075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.909306 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.909440 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.910298 (+ 858us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.910392 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
I20260924 10:40:06.911319 8194 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.8.60:33075: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f"
member_type: NON_VOTER
last_known_addr {
host: "127.2.8.60"
port: 33075
}
attrs {
promote: true
}
}
dest_uuid: "ee68668fd32841e2a0495725f909eee7"
cas_config_opid_index: 5
I20260924 10:40:06.911867 8194 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: NON_VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: true } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } }
I20260924 10:40:06.912900 8756 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 448a89e65a7c40c3b72df3b2ba71e68f (127.2.8.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: NON_VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: true } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } } }
I20260924 10:40:06.914520 8756 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.8.60:33075
I20260924 10:40:06.914680 8756 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ee68668fd32841e2a0495725f909eee7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: NON_VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: true } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } } }
I20260924 10:40:06.914769 8756 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:06.915553 8888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:06.916960 8888 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 1 FOLLOWER]: Refusing update from remote peer ee68668fd32841e2a0495725f909eee7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 10:40:06.917836 8755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:06.918001 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.913128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:37391 (local address 127.0.0.1:42286)
0924 10:40:06.913581 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.913593 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.913653 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.914087 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.914093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.914114 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:06.914190 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.914195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.915276 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.915280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.916458 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.916469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.917606 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.917610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.917701 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.917740 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.917815 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.917889 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385}
I20260924 10:40:06.918421 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.913362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42286 (local address 127.2.8.61:37391)
0924 10:40:06.913784 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.913788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.913800 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:40:06.913838 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.913840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.913891 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.913974 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:06.914312 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.915155 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.916596 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.917261 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.917380 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.917534 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.918225 (+ 691us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.918298 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355}
I20260924 10:40:06.918666 8768 master.cc:537] Master@127.2.8.60:33075 shutting down...
I20260924 10:40:06.918814 8806 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 1 LEARNER]: Refusing update from remote peer ee68668fd32841e2a0495725f909eee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 10:40:06.919569 8755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: NON_VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260924 10:40:06.921026 8181 consensus_peers.cc:487] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 -> Peer 448a89e65a7c40c3b72df3b2ba71e68f (127.2.8.60:33075): Couldn't send request to peer 448a89e65a7c40c3b72df3b2ba71e68f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 10:40:06.922158 8806 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 448a89e65a7c40c3b72df3b2ba71e68f (127.2.8.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: NON_VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: true } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } } }
I20260924 10:40:06.924276 8835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee68668fd32841e2a0495725f909eee7. Latest consensus state: current_term: 1 leader_uuid: "ee68668fd32841e2a0495725f909eee7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: NON_VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: true } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } } }
I20260924 10:40:06.924417 8835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: This master's current role is: LEARNER
I20260924 10:40:06.924520 8835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee68668fd32841e2a0495725f909eee7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: NON_VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: true } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } } }
I20260924 10:40:06.924614 8835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: This master's current role is: LEARNER
I20260924 10:40:06.925721 8754 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7: attempting to promote NON_VOTER 3192885cd0a94303817ce469f56c9ed2 to VOTER
I20260924 10:40:06.926108 8754 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } }
I20260924 10:40:06.927690 8806 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 1 LEARNER]: Refusing update from remote peer ee68668fd32841e2a0495725f909eee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260924 10:40:06.927927 8181 consensus_peers.cc:487] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 -> Peer 448a89e65a7c40c3b72df3b2ba71e68f (127.2.8.60:33075): Couldn't send request to peer 448a89e65a7c40c3b72df3b2ba71e68f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 10:40:06.928105 8768 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:40:06.928372 8768 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:06.928491 8768 tablet_replica.cc:330] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: stopping tablet replica
I20260924 10:40:06.928910 8944 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 10:40:06.930860 8755 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3192885cd0a94303817ce469f56c9ed2 (127.2.8.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } } }
I20260924 10:40:06.932516 8755 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ee68668fd32841e2a0495725f909eee7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } } }
I20260924 10:40:06.932636 8755 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:06.932940 8756 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee68668fd32841e2a0495725f909eee7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } } }
I20260924 10:40:06.933053 8756 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:06.933651 8806 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3192885cd0a94303817ce469f56c9ed2 (127.2.8.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } } }
I20260924 10:40:06.934827 8943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee68668fd32841e2a0495725f909eee7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } } }
I20260924 10:40:06.934957 8943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:40:06.944497 8768 master.cc:552] Master@127.2.8.60:33075 shutdown complete.
I20260924 10:40:06.944568 8768 master_runner.cc:305] Clearing existing system tablet
I20260924 10:40:06.944795 8768 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:06.944877 8768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:06.946089 8768 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:40:06.946920 8946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:06.947161 8768 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:06.947293 8768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:40:06.947338 8768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/wal
uuid: "448a89e65a7c40c3b72df3b2ba71e68f"
format_stamp: "Formatted at 2026-09-24 10:40:06 on dist-test-slave-j38h"
server_key: "809ffb3635f84b2798bf892607d53f28"
server_key_iv: "c8dfac7c6bcfe982e9b9e240b5bd27e3"
server_key_version: "encryptionkey@0"
I20260924 10:40:06.949090 8768 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:06.951220 8768 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 10:40:06.951326 8768 log.cc:1196] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 10:40:06.951654 8768 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: Deleting consensus metadata
I20260924 10:40:06.951864 8768 master_runner.cc:315] Copying system tablet from 127.2.8.62:38611
I20260924 10:40:06.953171 8768 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: tablet copy: Beginning tablet copy session from remote peer at address 127.2.8.62:38611
I20260924 10:40:06.961139 8952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.953790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.0.0.1:60936)
0924 10:40:06.954050 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:06.954065 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:06.954221 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:06.954490 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:06.954493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:06.954749 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:06.954836 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.954842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.955695 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.955700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.956285 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:06.956293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.956757 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.956774 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:06.956858 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.957459 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:06.957475 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:06.957831 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:06.959568 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.959573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.959576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.959776 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.960067 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:06.960072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:06.960074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:06.960161 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:06.960536 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:06.960539 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:06.960643 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 10:40:06.960748 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 10:40:06.960897 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":109,"threads_started":1}
I20260924 10:40:06.961172 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:06.954114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.2.8.62:38611)
0924 10:40:06.954251 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:06.954258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:06.954276 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:40:06.954320 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:06.954323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:06.954367 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:06.954460 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:06.954963 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.955586 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.956423 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:06.956658 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:06.956768 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:06.957969 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:06.957972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:06.957975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:06.958000 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:06.959452 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.959880 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.959885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.959887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.959968 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:06.960251 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:06.960256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:06.960257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:06.960440 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:06.960546 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:06.960802 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 10:40:06.960917 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:40:06.961951 8215 tablet_copy_service.cc:140] P ee68668fd32841e2a0495725f909eee7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 448a89e65a7c40c3b72df3b2ba71e68f ({username='oryx', principal='oryx/127.2.8.60@KRBTEST.COM'} at 127.0.0.1:60936)
I20260924 10:40:06.962064 8215 tablet_copy_service.cc:161] P ee68668fd32841e2a0495725f909eee7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 448a89e65a7c40c3b72df3b2ba71e68f at {username='oryx', principal='oryx/127.2.8.60@KRBTEST.COM'} at 127.0.0.1:60936: session id = 448a89e65a7c40c3b72df3b2ba71e68f-00000000000000000000000000000000
I20260924 10:40:06.963083 8215 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 10:40:06.964275 8768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 10:40:06.966261 8768 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: tablet copy: Starting download of 0 data blocks...
I20260924 10:40:06.966464 8768 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: tablet copy: Starting download of 1 WAL segments...
I20260924 10:40:06.967877 8768 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 10:40:06.969492 8215 tablet_copy_service.cc:342] P ee68668fd32841e2a0495725f909eee7: Request end of tablet copy session 448a89e65a7c40c3b72df3b2ba71e68f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.8.60@KRBTEST.COM'} at 127.0.0.1:60936
I20260924 10:40:06.969601 8215 tablet_copy_service.cc:433] P ee68668fd32841e2a0495725f909eee7: ending tablet copy session 448a89e65a7c40c3b72df3b2ba71e68f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 448a89e65a7c40c3b72df3b2ba71e68f
I20260924 10:40:06.976271 8768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:06.978189 8953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:06.978449 8956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:06.979017 8954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:06.981137 8768 server_base.cc:1034] running on GCE node
I20260924 10:40:06.981309 8768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:06.981630 8768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:06.982802 8768 hybrid_clock.cc:648] HybridClock initialized: now 1790246406982801 us; error 49 us; skew 500 ppm
Sep 24 10:40:06 dist-test-slave-j38h krb5kdc[8150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246406, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:06.984639 8768 init.cc:377] Logged in from keytab as oryx/127.2.8.60@KRBTEST.COM (short username oryx)
I20260924 10:40:06.985289 8768 webserver.cc:460] Webserver started at http://127.2.8.60:33023/ using document root <none> and password file <none>
I20260924 10:40:06.985519 8768 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:06.985585 8768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:06.987120 8768 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:40:06.988185 8963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:06.988446 8768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 10:40:06.988555 8768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/wal
uuid: "448a89e65a7c40c3b72df3b2ba71e68f"
format_stamp: "Formatted at 2026-09-24 10:40:06 on dist-test-slave-j38h"
server_key: "809ffb3635f84b2798bf892607d53f28"
server_key_iv: "c8dfac7c6bcfe982e9b9e240b5bd27e3"
server_key_version: "encryptionkey@0"
I20260924 10:40:06.988695 8768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:07.007963 8768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:07.011030 8768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:07.017267 8768 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.60:33075
I20260924 10:40:07.017275 9015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.60:33075 every 8 connection(s)
I20260924 10:40:07.017951 8768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/master-2/data/info.pb
I20260924 10:40:07.019623 9016 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:40:07.020606 9016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: Bootstrap starting.
I20260924 10:40:07.025345 8520 heartbeater.cc:503] Master 127.2.8.62:38611 was elected leader, sending a full tablet report...
I20260924 10:40:07.031251 9016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:40:07.031582 9016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: Bootstrap complete.
I20260924 10:40:07.032126 9016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } }
I20260924 10:40:07.032334 9016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 448a89e65a7c40c3b72df3b2ba71e68f, State: Initialized, Role: LEARNER
I20260924 10:40:07.032481 9016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } }
I20260924 10:40:07.032850 9019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } } }
I20260924 10:40:07.033082 9016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:40:07.033082 9019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [sys.catalog]: This master's current role is: LEARNER
I20260924 10:40:07.036865 9030 catalog_manager.cc:1259] Loaded cluster ID: 11926dce6cf44ca1acfbebd864657b29
I20260924 10:40:07.037017 9030 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: loading cluster ID for follower catalog manager: success
I20260924 10:40:07.039153 9030 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: acquiring CA information for follower catalog manager: success
I20260924 10:40:07.040464 9030 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 10:40:07.045882 8940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:07.046186 8940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:07.046250 8940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:07.052268 8940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:07.052340 8940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:07.052431 8940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:40:07.060508 8940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.1.keytab
--principal=oryx/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:42063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=34681
--tserver_master_addrs=127.2.8.62:38611,127.2.8.61:37391,127.2.8.60:33075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:07.061759 8940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:07.062981 8940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:07.074095 9036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:07.074150 9038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:07.074280 8940 server_base.cc:1034] running on GCE node
W20260924 10:40:07.074147 9035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:07.074734 8940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:07.075373 8940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:07.076557 8940 hybrid_clock.cc:648] HybridClock initialized: now 1790246407076523 us; error 52 us; skew 500 ppm
Sep 24 10:40:07 dist-test-slave-j38h krb5kdc[8150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246407, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:07.079491 8940 init.cc:377] Logged in from keytab as oryx/127.2.8.1@KRBTEST.COM (short username oryx)
I20260924 10:40:07.080358 8940 webserver.cc:460] Webserver started at http://127.2.8.1:34681/ using document root <none> and password file <none>
I20260924 10:40:07.080901 8940 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:07.081032 8940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:07.084801 8940 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260924 10:40:07.086930 9045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:07.088220 8940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:40:07.088351 8940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "0bd5a2a65f554fdc95fe660cf67364ff"
format_stamp: "Formatted at 2026-09-24 10:40:05 on dist-test-slave-j38h"
server_key: "1af27d0e914c671794cfdbabb941e70a"
server_key_iv: "0d5d2caad75459c90799af8fd2b075f4"
server_key_version: "encryptionkey@0"
I20260924 10:40:07.088908 8940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:07.105847 8940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:07.108964 8940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:07.109246 8940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:07.109849 8940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:07.110728 8940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:07.110800 8940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:07.110870 8940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:07.110912 8940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:07.119889 8940 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:42063
I20260924 10:40:07.119913 9158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:42063 every 8 connection(s)
I20260924 10:40:07.120954 8940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:40:07.128593 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 8940
I20260924 10:40:07.128803 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 8391
Sep 24 10:40:07 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246407, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.60@KRBTEST.COM
Sep 24 10:40:07 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246407, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.61@KRBTEST.COM
Sep 24 10:40:07 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246407, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM
I20260924 10:40:07.136009 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:35103
--local_ip_for_outbound_sockets=127.2.8.2
--tserver_master_addrs=127.2.8.62:38611
--webserver_port=40375
--webserver_interface=127.2.8.2
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.2.keytab
--principal=oryx/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.8.62:38611,127.2.8.61:37391
--tserver_master_addrs=127.2.8.62:38611,127.2.8.61:37391,127.2.8.60:33075 with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:40:07.139835 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.123499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:36285 (local address 127.2.8.62:38611)
0924 10:40:07.123634 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.123638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.124158 (+ 520us) server_negotiation.cc:407] Connection header received
0924 10:40:07.125237 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.125240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.125291 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.125354 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:07.126883 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.127417 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.130462 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.131349 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.131437 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.135647 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:07.135650 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:07.135653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:07.135681 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:07.137567 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.138033 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.138036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.138038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.138116 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.138442 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.138445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.138446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.138640 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:07.138729 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.139502 (+ 773us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.139629 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:40:07.140442 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.123509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:43713 (local address 127.2.8.61:37391)
0924 10:40:07.123733 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.123738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.124156 (+ 418us) server_negotiation.cc:407] Connection header received
0924 10:40:07.125234 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.125237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.125292 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.125346 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:07.126895 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.127671 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.129953 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.130370 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.130511 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.135110 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:07.135113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:07.135117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:07.135148 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:07.138392 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.138929 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.138932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.138934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.139056 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.139368 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.139374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.139379 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.139588 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:07.139701 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.140188 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.140308 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260924 10:40:07.140460 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.123626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:57629 (local address 127.2.8.60:33075)
0924 10:40:07.123900 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.123904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.124595 (+ 691us) server_negotiation.cc:407] Connection header received
0924 10:40:07.125208 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.125212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.125273 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.125382 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:07.126854 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.127894 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.129676 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.130035 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.130147 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.134704 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:07.134708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:07.134711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:07.134738 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:07.137107 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.137818 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.137823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.137826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.137913 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.138365 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.138371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.138376 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.138684 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:07.138768 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.139890 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.140190 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
I20260924 10:40:07.141463 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.123300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:37391 (local address 127.2.8.1:43713)
0924 10:40:07.124003 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:07.124057 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:07.124998 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:07.125557 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:07.125566 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:07.126078 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:07.126722 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.126729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.127798 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.127801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.129851 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.129858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.130640 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.130661 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.130796 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.131227 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:07.131248 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:07.134980 (+ 3732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:07.138531 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.138537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.138540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.138810 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.139168 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.139171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.139173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.139265 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.139759 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:07.139762 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:07.139846 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 10:40:07.140724 (+ 878us) client_negotiation.cc:240] Negotiation successful
0924 10:40:07.140857 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":131,"spinlock_wait_cycles":45312,"thread_start_us":131,"threads_started":1}
I20260924 10:40:07.141479 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.123300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.1:36285)
0924 10:40:07.124002 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:07.124057 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:07.124992 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:07.125557 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:07.125566 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:07.126080 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:07.126720 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.126729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.127556 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.127558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.130336 (+ 2778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.130344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.131584 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.131603 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.131692 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.132176 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:07.132192 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:07.135507 (+ 3315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:07.137686 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.137689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.137691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.137928 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.138258 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.138261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.138262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.138345 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.138797 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:07.138805 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:07.139208 (+ 403us) client_negotiation.cc:769] Sending connection context
0924 10:40:07.140085 (+ 877us) client_negotiation.cc:240] Negotiation successful
0924 10:40:07.140324 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":1674,"spinlock_wait_cycles":64256,"thread_start_us":132,"threads_started":1}
I20260924 10:40:07.141990 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.123300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:33075 (local address 127.2.8.1:57629)
0924 10:40:07.124498 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:07.124510 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:07.125017 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:07.125557 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:07.125572 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:07.126056 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:07.126711 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.126727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.128001 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.128006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.129532 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.129546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.130131 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.130187 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.130444 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.131085 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:07.131111 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:07.134531 (+ 3420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:07.137243 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.137249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.137268 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.137709 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.138016 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.138019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.138021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.138200 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.138803 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:07.138805 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:07.139207 (+ 402us) client_negotiation.cc:769] Sending connection context
0924 10:40:07.139439 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 10:40:07.141087 (+ 1648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"mutex_wait_us":1600,"spinlock_wait_cycles":12928,"thread_start_us":105,"threads_started":1}
I20260924 10:40:07.142796 9159 heartbeater.cc:347] Connected to a master server at 127.2.8.60:33075
I20260924 10:40:07.142807 9161 heartbeater.cc:347] Connected to a master server at 127.2.8.61:37391
I20260924 10:40:07.143060 9159 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:07.143059 9161 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:07.143087 9160 heartbeater.cc:347] Connected to a master server at 127.2.8.62:38611
I20260924 10:40:07.143218 9160 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:07.143710 9161 heartbeater.cc:511] Master 127.2.8.61:37391 requested a full tablet report, sending...
I20260924 10:40:07.143711 9160 heartbeater.cc:511] Master 127.2.8.62:38611 requested a full tablet report, sending...
I20260924 10:40:07.143710 9159 heartbeater.cc:511] Master 127.2.8.60:33075 requested a full tablet report, sending...
I20260924 10:40:07.145174 8194 ts_manager.cc:194] Re-registered known tserver with Master: 0bd5a2a65f554fdc95fe660cf67364ff (127.2.8.1:42063)
I20260924 10:40:07.145364 8796 ts_manager.cc:194] Registered new tserver with Master: 0bd5a2a65f554fdc95fe660cf67364ff (127.2.8.1:42063)
I20260924 10:40:07.145364 8980 ts_manager.cc:194] Registered new tserver with Master: 0bd5a2a65f554fdc95fe660cf67364ff (127.2.8.1:42063)
I20260924 10:40:07.145773 8194 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:36285
I20260924 10:40:07.237602 8654 heartbeater.cc:503] Master 127.2.8.62:38611 was elected leader, sending a full tablet report...
W20260924 10:40:07.252619 9167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:07.252895 9167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:07.252938 9167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:07.257081 9167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:07.257144 9167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:07.257201 9167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:40:07.262575 9167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.2.keytab
--principal=oryx/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:35103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=40375
--tserver_master_addrs=127.2.8.62:38611,127.2.8.61:37391,127.2.8.60:33075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:07.263675 9167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:07.264671 9167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:07.274150 9175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:07.274188 9173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:07.274163 9172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:07.274482 9167 server_base.cc:1034] running on GCE node
I20260924 10:40:07.274832 9167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:07.275388 9167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:07.276593 9167 hybrid_clock.cc:648] HybridClock initialized: now 1790246407276581 us; error 23 us; skew 500 ppm
Sep 24 10:40:07 dist-test-slave-j38h krb5kdc[8150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246407, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:07.279261 9167 init.cc:377] Logged in from keytab as oryx/127.2.8.2@KRBTEST.COM (short username oryx)
I20260924 10:40:07.280140 9167 webserver.cc:460] Webserver started at http://127.2.8.2:40375/ using document root <none> and password file <none>
I20260924 10:40:07.280696 9167 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:07.280771 9167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:07.284466 9167 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:40:07.286427 9182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:07.287671 9167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:40:07.287798 9167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "6b40ad03a8884554a439d5ac6838d033"
format_stamp: "Formatted at 2026-09-24 10:40:05 on dist-test-slave-j38h"
server_key: "9d1a96dc669b237b1c415e7ed030d529"
server_key_iv: "44710e2487f18837b0fda01da1d2040c"
server_key_version: "encryptionkey@0"
I20260924 10:40:07.288326 9167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:07.306105 9167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:07.308849 9167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:07.309078 9167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:07.309657 9167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:07.310488 9167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:07.310556 9167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:07.310626 9167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:07.310668 9167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:07.315294 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.311476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.2.8.60:33075)
0924 10:40:07.311641 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.311647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.311668 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:40:07.311766 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.311770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.311839 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.311966 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:07.312176 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.312971 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.314280 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.314784 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.314899 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.315056 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.315097 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.315165 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:40:07.315294 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.311342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:33075 (local address 127.0.0.1:49118)
0924 10:40:07.311589 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:07.311602 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:07.311678 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:07.311979 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:07.311981 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:07.311994 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:07.312075 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.312080 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.313112 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.313116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.314154 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.314164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.314913 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.314916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.315000 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.315021 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:40:07.315085 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 10:40:07.315161 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260924 10:40:07.315999 8982 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 10:40:07.316459 9019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee68668fd32841e2a0495725f909eee7. Latest consensus state: current_term: 1 leader_uuid: "ee68668fd32841e2a0495725f909eee7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } } }
I20260924 10:40:07.316612 9019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [sys.catalog]: This master's current role is: LEARNER
I20260924 10:40:07.321048 9167 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:35103
I20260924 10:40:07.321069 9295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:35103 every 8 connection(s)
I20260924 10:40:07.322099 9167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:40:07.323288 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 9167
I20260924 10:40:07.323453 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 8525
I20260924 10:40:07.331386 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:43511
--local_ip_for_outbound_sockets=127.2.8.3
--tserver_master_addrs=127.2.8.62:38611
--webserver_port=35585
--webserver_interface=127.2.8.3
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.3.keytab
--principal=oryx/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.8.62:38611,127.2.8.61:37391
--tserver_master_addrs=127.2.8.62:38611,127.2.8.61:37391,127.2.8.60:33075 with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:40:07 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246407, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM
Sep 24 10:40:07 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246407, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.60@KRBTEST.COM
Sep 24 10:40:07 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246407, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.61@KRBTEST.COM
I20260924 10:40:07.338721 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.324183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:56253 (local address 127.2.8.62:38611)
0924 10:40:07.324363 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.324368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.325439 (+ 1071us) server_negotiation.cc:407] Connection header received
0924 10:40:07.326782 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.326786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.326840 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.326913 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:07.328515 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.329081 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.330823 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.331146 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.331221 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.334497 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:07.334502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:07.334506 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:07.334534 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:07.336165 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.336808 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.336812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.336813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.336897 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.337316 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.337321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.337323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.337563 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:07.337638 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.338454 (+ 816us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.338563 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 10:40:07.339776 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.324291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.2:56253)
0924 10:40:07.325267 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:07.325323 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:07.326572 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:07.327092 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:07.327103 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:07.327642 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:07.328348 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.328370 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.329220 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.329226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.330695 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.330708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.331361 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.331420 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.331611 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.332252 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:07.332271 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:07.334292 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:07.336278 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.336284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.336299 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.336704 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.337042 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.337046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.337048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.337209 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.337728 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:07.337735 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:07.338109 (+ 374us) client_negotiation.cc:769] Sending connection context
0924 10:40:07.338395 (+ 286us) client_negotiation.cc:240] Negotiation successful
0924 10:40:07.338759 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":541,"threads_started":1}
I20260924 10:40:07.340886 9297 heartbeater.cc:347] Connected to a master server at 127.2.8.62:38611
I20260924 10:40:07.341150 9297 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:07.341626 9297 heartbeater.cc:511] Master 127.2.8.62:38611 requested a full tablet report, sending...
I20260924 10:40:07.342561 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.330063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:53527 (local address 127.2.8.61:37391)
0924 10:40:07.330192 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.330198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.330755 (+ 557us) server_negotiation.cc:407] Connection header received
0924 10:40:07.330930 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.330933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.330977 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.331024 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:07.331594 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.332355 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.333558 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.333920 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.333991 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.337173 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:07.337179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:07.337185 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:07.337216 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:07.339301 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.339771 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.339776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.339777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.339855 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.340461 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.340464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.340466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.340664 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:07.340731 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.342337 (+ 1606us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.342451 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:40:07.342929 8194 ts_manager.cc:194] Re-registered known tserver with Master: 6b40ad03a8884554a439d5ac6838d033 (127.2.8.2:35103)
I20260924 10:40:07.343206 9301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.325749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:33075 (local address 127.2.8.2:50821)
0924 10:40:07.326028 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:07.326040 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:07.326582 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:07.330108 (+ 3526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:07.330111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:07.330312 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:07.330442 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.330450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.331713 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.331720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.332729 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.332743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.333549 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.333567 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.333685 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.334436 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:07.334464 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:07.336970 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:07.341130 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.341136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.341141 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.341417 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.341775 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.341778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.341781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.341883 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.342398 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:07.342404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:07.342539 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 10:40:07.342774 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 10:40:07.342941 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":136,"threads_started":1}
I20260924 10:40:07.343616 8194 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:56253
I20260924 10:40:07.343849 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.325798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:50821 (local address 127.2.8.60:33075)
0924 10:40:07.329179 (+ 3381us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.329186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.329204 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:40:07.329246 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.329252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.329303 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.329385 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:07.330579 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.331314 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.332897 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.333289 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.333397 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.337130 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:07.337133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:07.337136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:07.337160 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:07.340529 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.341562 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.341565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.341567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.341660 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.341996 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.342003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.342007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.342245 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:07.342349 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.343494 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.343633 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3225}
I20260924 10:40:07.344192 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.329726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:37391 (local address 127.2.8.2:53527)
0924 10:40:07.330675 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:07.330687 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:07.330820 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:07.331155 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:07.331157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:07.331345 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:07.331464 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.331471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.332496 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.332503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.333420 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.333430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.334160 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.334199 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.334292 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.334750 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:07.334997 (+ 247us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:07.337013 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:07.339422 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.339425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.339427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.339646 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.339983 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.339987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.339989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.340093 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.340806 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:07.340808 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:07.342143 (+ 1335us) client_negotiation.cc:769] Sending connection context
0924 10:40:07.343870 (+ 1727us) client_negotiation.cc:240] Negotiation successful
0924 10:40:07.344037 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"thread_start_us":86,"threads_started":1}
I20260924 10:40:07.344655 9296 heartbeater.cc:347] Connected to a master server at 127.2.8.60:33075
I20260924 10:40:07.344736 9296 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:07.345335 9298 heartbeater.cc:347] Connected to a master server at 127.2.8.61:37391
I20260924 10:40:07.345434 9298 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:07.345696 9298 heartbeater.cc:511] Master 127.2.8.61:37391 requested a full tablet report, sending...
I20260924 10:40:07.345695 9296 heartbeater.cc:511] Master 127.2.8.60:33075 requested a full tablet report, sending...
I20260924 10:40:07.346472 8980 ts_manager.cc:194] Registered new tserver with Master: 6b40ad03a8884554a439d5ac6838d033 (127.2.8.2:35103)
I20260924 10:40:07.347033 8796 ts_manager.cc:194] Registered new tserver with Master: 6b40ad03a8884554a439d5ac6838d033 (127.2.8.2:35103)
W20260924 10:40:07.445443 9303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:07.445727 9303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:07.445818 9303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:07.449841 9303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:07.449915 9303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:07.450007 9303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:40:07.455305 9303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.3.keytab
--principal=oryx/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:43511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=35585
--tserver_master_addrs=127.2.8.62:38611,127.2.8.61:37391,127.2.8.60:33075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:07.456377 9303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:07.457472 9303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:07.467002 9309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:07.467072 9303 server_base.cc:1034] running on GCE node
W20260924 10:40:07.466995 9311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:07.467015 9308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:07.467679 9303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:07.468281 9303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:07.469478 9303 hybrid_clock.cc:648] HybridClock initialized: now 1790246407469446 us; error 41 us; skew 500 ppm
Sep 24 10:40:07 dist-test-slave-j38h krb5kdc[8150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246407, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:07.472196 9303 init.cc:377] Logged in from keytab as oryx/127.2.8.3@KRBTEST.COM (short username oryx)
I20260924 10:40:07.473084 9303 webserver.cc:460] Webserver started at http://127.2.8.3:35585/ using document root <none> and password file <none>
I20260924 10:40:07.473641 9303 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:07.473747 9303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:07.477473 9303 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 10:40:07.479597 9318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:07.481024 9303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:40:07.481124 9303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a"
format_stamp: "Formatted at 2026-09-24 10:40:06 on dist-test-slave-j38h"
server_key: "7e5fe8e803715548b2a5f63f87cc87bd"
server_key_iv: "8aacb5352099fd06385f6070e8c7cc96"
server_key_version: "encryptionkey@0"
I20260924 10:40:07.481753 9303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:07.522068 9303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:07.524964 9303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:07.525187 9303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:07.525784 9303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:07.526631 9303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:07.526701 9303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:07.526772 9303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:07.526813 9303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:07.535802 9303 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:43511
I20260924 10:40:07.535851 9431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:43511 every 8 connection(s)
I20260924 10:40:07.536888 9303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:40:07.538834 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 9303
Sep 24 10:40:07 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246407, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM
I20260924 10:40:07.551887 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.540928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42294 (local address 127.2.8.61:37391)
0924 10:40:07.542084 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.542089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.542102 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:40:07.542145 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.542150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.542209 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.542296 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:07.543257 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.544026 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.545246 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.545577 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.545652 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.547484 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:07.547488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:07.547490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:07.547515 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:07.549449 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.550030 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.550034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.550036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.550119 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.550424 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.550430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.550435 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.550657 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:07.550788 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.551525 (+ 737us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.551658 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097,"thread_start_us":97,"threads_started":1}
Sep 24 10:40:07 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246407, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.61@KRBTEST.COM
Sep 24 10:40:07 dist-test-slave-j38h krb5kdc[8150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246407, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.60@KRBTEST.COM
I20260924 10:40:07.556329 8234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.538737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:58815 (local address 127.2.8.62:38611)
0924 10:40:07.538891 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.538896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.541017 (+ 2121us) server_negotiation.cc:407] Connection header received
0924 10:40:07.542245 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.542248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.542317 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.542394 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:07.544085 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.544873 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.546602 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.546998 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.547085 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.550476 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:07.550481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:07.550484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:07.550509 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:07.552363 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.553016 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.553019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.553023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.553107 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.554427 (+ 1320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.554432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.554434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.554698 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:07.554796 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.556049 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.556171 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:40:07.557235 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.541243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.2.8.60:33075)
0924 10:40:07.541822 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.541826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.542854 (+ 1028us) server_negotiation.cc:407] Connection header received
0924 10:40:07.543136 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.543139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.543189 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.543258 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:07.545298 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.546091 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.549218 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.549577 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.549645 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.552992 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:07.552996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:07.552999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:07.553028 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:07.555078 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.555654 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.555657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.555659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.555737 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.556064 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.556067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.556069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.556263 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:07.556365 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.556944 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.557088 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":79,"threads_started":1}
I20260924 10:40:07.557650 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.540920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.2.8.62:38611)
0924 10:40:07.541960 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.541964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.542650 (+ 686us) server_negotiation.cc:407] Connection header received
0924 10:40:07.544216 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.544220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.544268 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.544328 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:07.547276 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.548049 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.550149 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.550519 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.550583 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.553460 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:07.553464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:07.553467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:07.553495 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:07.555206 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.555724 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.555733 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.555737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.555822 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.556177 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.556180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.556182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.556425 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:07.556568 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.557238 (+ 670us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.557367 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":972,"thread_start_us":87,"threads_started":1}
I20260924 10:40:07.558058 9436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.539245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.3:58815)
0924 10:40:07.540834 (+ 1589us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:07.540891 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:07.542021 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:07.542735 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:07.542747 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:07.543253 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:07.543921 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.543939 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.545055 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.545058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.546463 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.546473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.547230 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.547287 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.547518 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.548309 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:07.548333 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:07.550278 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:07.552499 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.552505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.552521 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.552870 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.553218 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.553221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.553223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.553674 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.555344 (+ 1670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:07.555353 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:07.555723 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 10:40:07.556550 (+ 827us) client_negotiation.cc:240] Negotiation successful
0924 10:40:07.556913 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1385,"mutex_wait_us":26,"thread_start_us":144,"threads_started":1}
I20260924 10:40:07.559317 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.540829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:44141 (local address 127.2.8.61:37391)
0924 10:40:07.540959 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.540965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.542511 (+ 1546us) server_negotiation.cc:407] Connection header received
0924 10:40:07.542683 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.542686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.542729 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.542777 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:07.544081 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.544848 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.549880 (+ 5032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.550470 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.550553 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.555501 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:07.555505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:07.555510 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:07.555538 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:07.557250 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.557717 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.557723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.557726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.557808 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.558124 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.558128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.558129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.558343 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:07.558440 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.558803 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.558908 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:40:07.559856 9437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.539245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:37391 (local address 127.2.8.3:44141)
0924 10:40:07.542429 (+ 3184us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:07.542443 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:07.542572 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:07.542926 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:07.542929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:07.543493 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:07.543923 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.543939 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.545050 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.545057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.547143 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.547151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.550692 (+ 3541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.550972 (+ 280us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.551307 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.552134 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:07.552152 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:07.554071 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:07.557375 (+ 3304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.557377 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.557379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.557603 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.557914 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.557916 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.557918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.558024 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.558467 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:07.558469 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:07.558572 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 10:40:07.559496 (+ 924us) client_negotiation.cc:240] Negotiation successful
0924 10:40:07.559647 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3099,"mutex_wait_us":391,"spinlock_wait_cycles":53632,"thread_start_us":178,"threads_started":1}
I20260924 10:40:07.560356 9433 heartbeater.cc:347] Connected to a master server at 127.2.8.62:38611
I20260924 10:40:07.560613 9433 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:07.560700 9434 heartbeater.cc:347] Connected to a master server at 127.2.8.61:37391
I20260924 10:40:07.560755 9434 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:07.561244 9434 heartbeater.cc:511] Master 127.2.8.61:37391 requested a full tablet report, sending...
I20260924 10:40:07.561244 9433 heartbeater.cc:511] Master 127.2.8.62:38611 requested a full tablet report, sending...
I20260924 10:40:07.562287 8195 ts_manager.cc:194] Re-registered known tserver with Master: ad2b0cf23c7c40fd8509b8438ea59d1a (127.2.8.3:43511)
I20260924 10:40:07.562356 8796 ts_manager.cc:194] Registered new tserver with Master: ad2b0cf23c7c40fd8509b8438ea59d1a (127.2.8.3:43511)
I20260924 10:40:07.562462 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.540353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:45621 (local address 127.2.8.60:33075)
0924 10:40:07.540486 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.540491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.543416 (+ 2925us) server_negotiation.cc:407] Connection header received
0924 10:40:07.545403 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.545406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.545455 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.545505 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:07.547266 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.548738 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.551543 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.551901 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.551995 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.558739 (+ 6744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:07.558743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:07.558746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:07.558771 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:07.560542 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.561224 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.561227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.561229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.561310 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.561593 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.561596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.561598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.561769 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:07.561878 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.562203 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.562320 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:40:07.562916 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.540022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:33075 (local address 127.2.8.3:45621)
0924 10:40:07.543337 (+ 3315us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:07.543351 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:07.543629 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:07.546488 (+ 2859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:07.546491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:07.546682 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:07.546794 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.546801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.548866 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.548869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.549753 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.549764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.553494 (+ 3730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.553511 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.553618 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.554220 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:07.554231 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:07.558359 (+ 4128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:07.560827 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.560830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.560831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.561091 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.561386 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:07.561391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:07.561393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:07.561478 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:07.561919 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:07.561922 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:07.562029 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 10:40:07.562616 (+ 587us) client_negotiation.cc:240] Negotiation successful
0924 10:40:07.562749 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3251,"spinlock_wait_cycles":14720,"thread_start_us":85,"threads_started":1}
I20260924 10:40:07.563009 8195 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:58815
I20260924 10:40:07.562192 8194 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 10:40:07.565835 9432 heartbeater.cc:347] Connected to a master server at 127.2.8.60:33075
I20260924 10:40:07.565927 9432 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:07.566107 9432 heartbeater.cc:511] Master 127.2.8.60:33075 requested a full tablet report, sending...
W20260924 10:40:07.566143 8194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:40:07.566834 8980 ts_manager.cc:194] Registered new tserver with Master: ad2b0cf23c7c40fd8509b8438ea59d1a (127.2.8.3:43511)
I20260924 10:40:07.568764 8944 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee68668fd32841e2a0495725f909eee7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: NON_VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: true } } }
I20260924 10:40:07.568915 8944 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:07.572196 8944 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7: attempting to promote NON_VOTER 448a89e65a7c40c3b72df3b2ba71e68f to VOTER
I20260924 10:40:07.572515 8944 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } }
I20260924 10:40:07.573544 8982 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 1 LEARNER]: Refusing update from remote peer ee68668fd32841e2a0495725f909eee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 10:40:07.573637 8806 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 1 FOLLOWER]: Refusing update from remote peer ee68668fd32841e2a0495725f909eee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 10:40:07.573932 8944 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 10:40:07.574260 8944 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 10:40:07.575947 8944 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 448a89e65a7c40c3b72df3b2ba71e68f (127.2.8.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } } }
I20260924 10:40:07.576668 8806 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 448a89e65a7c40c3b72df3b2ba71e68f (127.2.8.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } } }
I20260924 10:40:07.576522 8982 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 448a89e65a7c40c3b72df3b2ba71e68f (127.2.8.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } } }
I20260924 10:40:07.577368 8944 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ee68668fd32841e2a0495725f909eee7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } } }
I20260924 10:40:07.577481 8944 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:07.577648 8944 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee68668fd32841e2a0495725f909eee7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } } }
I20260924 10:40:07.577739 8944 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:07.577881 8944 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee68668fd32841e2a0495725f909eee7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } } }
I20260924 10:40:07.577971 8944 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:07.578665 9019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee68668fd32841e2a0495725f909eee7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } } }
I20260924 10:40:07.578802 9019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:40:07.578789 9460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee68668fd32841e2a0495725f909eee7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } } }
I20260924 10:40:07.578918 9460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:40:07.585906 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.580767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:42063 (local address 127.0.0.1:56856)
0924 10:40:07.580915 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:07.580930 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:07.581030 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:07.582047 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:07.582050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:07.582067 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:07.582175 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.582181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.583494 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.583497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.584404 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.584412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.585516 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.585522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.585611 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.585634 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:40:07.585703 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 10:40:07.585768 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":22}
I20260924 10:40:07.585983 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.580924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.2.8.1:42063)
0924 10:40:07.581404 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.581418 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.581442 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:40:07.581524 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.581530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.581757 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.582106 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:07.582263 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.583370 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.584528 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.585378 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.585495 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.585640 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.585730 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.585833 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":140,"threads_started":1}
I20260924 10:40:07.586603 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.581375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:43511 (local address 127.0.0.1:42064)
0924 10:40:07.581681 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:07.581695 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:07.581766 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:07.582574 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:07.582576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:07.582593 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:07.582671 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.582675 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.583780 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.583783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.585023 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.585033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.586148 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.586154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.586267 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.586297 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:40:07.586337 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 10:40:07.586437 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":22,"thread_start_us":108,"threads_started":1}
I20260924 10:40:07.587258 9462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.581189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:35103 (local address 127.0.0.1:56706)
0924 10:40:07.581404 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:07.581420 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:07.581509 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:07.582961 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:07.582964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:07.582980 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:07.583057 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.583062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.584406 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.584409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.585437 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.585470 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.586982 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.586986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.587070 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.587087 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:40:07.587127 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 10:40:07.587172 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":95,"threads_started":1}
I20260924 10:40:07.587271 9465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.581763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.2.8.3:43511)
0924 10:40:07.582112 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.582123 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.582143 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:40:07.582212 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.582220 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.582425 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.582791 (+ 366us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:07.582888 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.583662 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.585153 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.586002 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.586835 (+ 833us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.587006 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.587053 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.587138 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":91,"threads_started":1}
I20260924 10:40:07.589069 9464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.581578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.2.8.2:35103)
0924 10:40:07.582520 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.582531 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.582552 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:40:07.582620 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.582625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.582803 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.583177 (+ 374us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:07.583286 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.584279 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.585595 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.586830 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.587542 (+ 712us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.587665 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.587710 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.587813 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":82,"threads_started":1}
I20260924 10:40:07.589052 9093 tablet_service.cc:1467] Processing CreateTablet for tablet 8d9a4ebc7dea4d5a823238cb404222e0 (DEFAULT_TABLE table=test-table [id=6a09339859e848d99bd4f24578b1a3a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:07.590091 9093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d9a4ebc7dea4d5a823238cb404222e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:07.589857 9366 tablet_service.cc:1467] Processing CreateTablet for tablet 8d9a4ebc7dea4d5a823238cb404222e0 (DEFAULT_TABLE table=test-table [id=6a09339859e848d99bd4f24578b1a3a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:07.590793 9366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d9a4ebc7dea4d5a823238cb404222e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:07.595747 9230 tablet_service.cc:1467] Processing CreateTablet for tablet 8d9a4ebc7dea4d5a823238cb404222e0 (DEFAULT_TABLE table=test-table [id=6a09339859e848d99bd4f24578b1a3a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:07.596746 9466 tablet_bootstrap.cc:492] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff: Bootstrap starting.
I20260924 10:40:07.596778 9230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d9a4ebc7dea4d5a823238cb404222e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:07.598861 9466 tablet_bootstrap.cc:654] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:07.598856 9467 tablet_bootstrap.cc:492] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a: Bootstrap starting.
I20260924 10:40:07.599500 9466 log.cc:824] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:07.600927 9467 tablet_bootstrap.cc:654] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:07.601392 9466 tablet_bootstrap.cc:492] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff: No bootstrap required, opened a new log
I20260924 10:40:07.601599 9466 ts_tablet_manager.cc:1397] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:40:07.601738 9467 log.cc:824] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:07.603669 9467 tablet_bootstrap.cc:492] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a: No bootstrap required, opened a new log
I20260924 10:40:07.603847 9467 ts_tablet_manager.cc:1397] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:40:07.604492 9470 tablet_bootstrap.cc:492] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033: Bootstrap starting.
I20260924 10:40:07.605677 9466 raft_consensus.cc:348] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } } peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } }
I20260924 10:40:07.605927 9466 raft_consensus.cc:374] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:07.605984 9466 raft_consensus.cc:729] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bd5a2a65f554fdc95fe660cf67364ff, State: Initialized, Role: FOLLOWER
I20260924 10:40:07.606446 9466 consensus_queue.cc:260] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } } peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } }
I20260924 10:40:07.606736 9470 tablet_bootstrap.cc:654] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:07.607419 9466 ts_tablet_manager.cc:1428] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:40:07.607482 9470 log.cc:824] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:07.607802 9160 heartbeater.cc:503] Master 127.2.8.62:38611 was elected leader, sending a full tablet report...
I20260924 10:40:07.607844 9467 raft_consensus.cc:348] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } } peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } }
I20260924 10:40:07.608093 9467 raft_consensus.cc:374] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:07.608147 9467 raft_consensus.cc:729] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2b0cf23c7c40fd8509b8438ea59d1a, State: Initialized, Role: FOLLOWER
I20260924 10:40:07.608675 9467 consensus_queue.cc:260] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } } peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } }
I20260924 10:40:07.609592 9467 ts_tablet_manager.cc:1428] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:40:07.610594 9433 heartbeater.cc:503] Master 127.2.8.62:38611 was elected leader, sending a full tablet report...
I20260924 10:40:07.615144 9470 tablet_bootstrap.cc:492] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033: No bootstrap required, opened a new log
I20260924 10:40:07.615350 9470 ts_tablet_manager.cc:1397] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260924 10:40:07.618741 9470 raft_consensus.cc:348] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } } peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } }
I20260924 10:40:07.618980 9470 raft_consensus.cc:374] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:07.619038 9470 raft_consensus.cc:729] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b40ad03a8884554a439d5ac6838d033, State: Initialized, Role: FOLLOWER
I20260924 10:40:07.619426 9470 consensus_queue.cc:260] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } } peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } }
I20260924 10:40:07.620085 9297 heartbeater.cc:503] Master 127.2.8.62:38611 was elected leader, sending a full tablet report...
I20260924 10:40:07.620419 9470 ts_tablet_manager.cc:1428] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260924 10:40:07.622336 9162 tablet.cc:2369] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:40:07.788730 9435 tablet.cc:2369] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:40:07.823585 9299 tablet.cc:2369] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:40:07.837189 9473 raft_consensus.cc:482] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:40:07.837502 9473 raft_consensus.cc:504] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } } peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } }
I20260924 10:40:07.838851 9473 leader_election.cc:288] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bd5a2a65f554fdc95fe660cf67364ff (127.2.8.1:42063), 6b40ad03a8884554a439d5ac6838d033 (127.2.8.2:35103)
I20260924 10:40:07.842815 9464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.839249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:35465 (local address 127.2.8.2:35103)
0924 10:40:07.839376 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.839381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.839468 (+ 87us) server_negotiation.cc:407] Connection header received
0924 10:40:07.839625 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.839630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.839686 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.839763 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:07.840173 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.840685 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.841852 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.842246 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.842364 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.842428 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.842604 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.842668 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:40:07.842820 9461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.839335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:41165 (local address 127.2.8.1:42063)
0924 10:40:07.839467 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.839471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.839609 (+ 138us) server_negotiation.cc:407] Connection header received
0924 10:40:07.839715 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.839717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.839786 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.839856 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:07.840203 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.840801 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.841852 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.842281 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.842398 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.842456 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.842629 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.842694 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:40:07.842842 9437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.839229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:42063 (local address 127.2.8.3:41165)
0924 10:40:07.839507 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:07.839522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:07.839626 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:07.839888 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:07.839890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:07.839906 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:07.840111 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.840120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.840914 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.840920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.841725 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.841734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.842398 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.842402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.842503 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.842529 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 10:40:07.842615 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 10:40:07.842683 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":49}
I20260924 10:40:07.842846 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.839185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:35103 (local address 127.2.8.3:35465)
0924 10:40:07.839378 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:07.839395 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:07.839507 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:07.839797 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:07.839800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:07.839821 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:07.840035 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.840044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.840812 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.840818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.841716 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:07.841725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.842366 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.842369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:07.842471 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.842497 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 10:40:07.842591 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 10:40:07.842663 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":35}
I20260924 10:40:07.843516 9113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9a4ebc7dea4d5a823238cb404222e0" candidate_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" is_pre_election: true
I20260924 10:40:07.843816 9113 raft_consensus.cc:2454] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2b0cf23c7c40fd8509b8438ea59d1a in term 0.
I20260924 10:40:07.844278 9322 leader_election.cc:302] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bd5a2a65f554fdc95fe660cf67364ff, ad2b0cf23c7c40fd8509b8438ea59d1a; no voters:
I20260924 10:40:07.844298 9250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9a4ebc7dea4d5a823238cb404222e0" candidate_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b40ad03a8884554a439d5ac6838d033" is_pre_election: true
I20260924 10:40:07.844565 9250 raft_consensus.cc:2454] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2b0cf23c7c40fd8509b8438ea59d1a in term 0.
I20260924 10:40:07.844595 9473 raft_consensus.cc:2781] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:40:07.844662 9473 raft_consensus.cc:482] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:40:07.844702 9473 raft_consensus.cc:3037] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:07.845837 9473 raft_consensus.cc:504] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } } peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } }
I20260924 10:40:07.846134 9473 leader_election.cc:288] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [CANDIDATE]: Term 1 election: Requested vote from peers 0bd5a2a65f554fdc95fe660cf67364ff (127.2.8.1:42063), 6b40ad03a8884554a439d5ac6838d033 (127.2.8.2:35103)
I20260924 10:40:07.846501 9113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9a4ebc7dea4d5a823238cb404222e0" candidate_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bd5a2a65f554fdc95fe660cf67364ff"
I20260924 10:40:07.846632 9113 raft_consensus.cc:3037] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:07.846782 9250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d9a4ebc7dea4d5a823238cb404222e0" candidate_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b40ad03a8884554a439d5ac6838d033"
I20260924 10:40:07.847020 9250 raft_consensus.cc:3037] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:07.847757 9113 raft_consensus.cc:2454] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2b0cf23c7c40fd8509b8438ea59d1a in term 1.
I20260924 10:40:07.848068 9322 leader_election.cc:302] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bd5a2a65f554fdc95fe660cf67364ff, ad2b0cf23c7c40fd8509b8438ea59d1a; no voters:
I20260924 10:40:07.848101 9250 raft_consensus.cc:2454] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2b0cf23c7c40fd8509b8438ea59d1a in term 1.
I20260924 10:40:07.848256 9473 raft_consensus.cc:2781] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:07.848526 9473 raft_consensus.cc:686] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [term 1 LEADER]: Becoming Leader. State: Replica: ad2b0cf23c7c40fd8509b8438ea59d1a, State: Running, Role: LEADER
I20260924 10:40:07.848954 9473 consensus_queue.cc:237] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } } peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } }
I20260924 10:40:07.852875 8194 catalog_manager.cc:5495] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad2b0cf23c7c40fd8509b8438ea59d1a (127.2.8.3). New cstate: current_term: 1 leader_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } health_report { overall_health: HEALTHY } } }
I20260924 10:40:07.882328 9465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.878789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42074 (local address 127.2.8.3:43511)
0924 10:40:07.878914 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.878918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.878933 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:40:07.879065 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.879068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.879122 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.879230 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:40:07.879455 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.879954 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.880818 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.881132 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.881243 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.881545 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:40:07.881954 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:40:07.882087 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.882126 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.882180 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 10:40:07.888347 9250 raft_consensus.cc:1261] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033 [term 1 FOLLOWER]: Refusing update from remote peer ad2b0cf23c7c40fd8509b8438ea59d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:07.888347 9113 raft_consensus.cc:1261] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff [term 1 FOLLOWER]: Refusing update from remote peer ad2b0cf23c7c40fd8509b8438ea59d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:07.889171 9473 consensus_queue.cc:1035] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:07.889410 9475 consensus_queue.cc:1035] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:07.897117 9478 mvcc.cc:205] Tried to move back new op lower bound from 7332849286702583808 to 7332849286552129536. Current Snapshot: MvccSnapshot[applied={T|T < 7332849286702583808}]
I20260924 10:40:07.900074 9480 mvcc.cc:205] Tried to move back new op lower bound from 7332849286702583808 to 7332849286552129536. Current Snapshot: MvccSnapshot[applied={T|T < 7332849286702583808}]
I20260924 10:40:07.905938 9479 mvcc.cc:205] Tried to move back new op lower bound from 7332849286702583808 to 7332849286552129536. Current Snapshot: MvccSnapshot[applied={T|T < 7332849286702583808}]
I20260924 10:40:07.908859 8195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:40:07.909045 8195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:40:07.912767 8195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee68668fd32841e2a0495725f909eee7: Sending DeleteTablet for 3 replicas of tablet 8d9a4ebc7dea4d5a823238cb404222e0
I20260924 10:40:07.913416 9093 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d9a4ebc7dea4d5a823238cb404222e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:07 UTC) from {username='oryx', principal='oryx/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:56856
I20260924 10:40:07.913684 9366 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d9a4ebc7dea4d5a823238cb404222e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:07 UTC) from {username='oryx', principal='oryx/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:42064
I20260924 10:40:07.913841 9230 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d9a4ebc7dea4d5a823238cb404222e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:07 UTC) from {username='oryx', principal='oryx/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:56706
I20260924 10:40:07.914167 9488 tablet_replica.cc:330] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a: stopping tablet replica
I20260924 10:40:07.914528 9488 raft_consensus.cc:2229] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:40:07.914937 9488 raft_consensus.cc:2258] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:07.915512 9489 tablet_replica.cc:330] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033: stopping tablet replica
I20260924 10:40:07.915786 9489 raft_consensus.cc:2229] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:40:07.916097 9489 raft_consensus.cc:2258] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:07.916524 9488 ts_tablet_manager.cc:1905] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:07.917047 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 8163
I20260924 10:40:07.918120 9489 ts_tablet_manager.cc:1905] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:07.920432 9488 ts_tablet_manager.cc:1918] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:40:07.920598 9488 log.cc:1196] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/8d9a4ebc7dea4d5a823238cb404222e0
I20260924 10:40:07.920818 9489 ts_tablet_manager.cc:1918] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:40:07.920940 9489 log.cc:1196] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/8d9a4ebc7dea4d5a823238cb404222e0
I20260924 10:40:07.921159 9488 ts_tablet_manager.cc:1939] T 8d9a4ebc7dea4d5a823238cb404222e0 P ad2b0cf23c7c40fd8509b8438ea59d1a: Deleting consensus metadata
I20260924 10:40:07.921285 9489 ts_tablet_manager.cc:1939] T 8d9a4ebc7dea4d5a823238cb404222e0 P 6b40ad03a8884554a439d5ac6838d033: Deleting consensus metadata
I20260924 10:40:07.922439 9487 tablet_replica.cc:330] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff: stopping tablet replica
I20260924 10:40:07.922806 9487 raft_consensus.cc:2229] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:40:07.923110 9487 raft_consensus.cc:2258] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 10:40:07.924335 9319 connection.cc:535] server connection from 127.0.0.1:42064 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:40:07.924480 9487 ts_tablet_manager.cc:1905] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 10:40:07.924654 9183 connection.cc:535] server connection from 127.0.0.1:56706 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:40:07.927232 9487 ts_tablet_manager.cc:1918] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:40:07.927374 9487 log.cc:1196] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-0/wal/wals/8d9a4ebc7dea4d5a823238cb404222e0
I20260924 10:40:07.927783 9487 ts_tablet_manager.cc:1939] T 8d9a4ebc7dea4d5a823238cb404222e0 P 0bd5a2a65f554fdc95fe660cf67364ff: Deleting consensus metadata
W20260924 10:40:07.928696 9046 connection.cc:441] server connection from 127.0.0.1:56856 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56856 (request call id 1) could send its response
I20260924 10:40:07.936389 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.916640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42302 (local address 127.2.8.61:37391)
0924 10:40:07.916742 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.916746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.925848 (+ 9102us) server_negotiation.cc:407] Connection header received
0924 10:40:07.926036 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.926039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.926086 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.926137 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:07.927137 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.927940 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.929094 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.929432 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.929541 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.931311 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:07.931316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:07.931319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:07.931345 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:07.933156 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.933616 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.933619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.933621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.933707 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.934020 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.934023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.934025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.934247 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:07.934801 (+ 554us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.935625 (+ 824us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.935733 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 10:40:07.940853 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.929918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.2.8.60:33075)
0924 10:40:07.930019 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.930023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.931496 (+ 1473us) server_negotiation.cc:407] Connection header received
0924 10:40:07.931682 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.931686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.931755 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.931809 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:07.932813 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.933649 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.934909 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.935257 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.935335 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.937287 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:07.937291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:07.937293 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:07.937318 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:07.938998 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.939484 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.939489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.939491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.939572 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.939885 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.939889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.939890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.940126 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:07.940241 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.940539 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.940656 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":40}
I20260924 10:40:07.942102 9444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.929381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.2.8.61:37391)
0924 10:40:07.929541 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.929545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.929837 (+ 292us) server_negotiation.cc:407] Connection header received
0924 10:40:07.930192 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.930196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.930239 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.930284 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:07.931333 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.932123 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.933396 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.933776 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.933865 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.935775 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:07.935778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:07.935781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:07.935804 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:07.938463 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.940537 (+ 2074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.940541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.940543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.940621 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.940900 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.940903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.940905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.941182 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:07.941278 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.941832 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.941957 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 10:40:07.945601 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:07.915718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.2.8.60:33075)
0924 10:40:07.915822 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:07.915838 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:07.935377 (+ 19539us) server_negotiation.cc:407] Connection header received
0924 10:40:07.935567 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:07.935571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:07.935615 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:07.935678 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:07.938011 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.938807 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.940027 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:07.940383 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:07.940495 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:07.942391 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:07.942394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:07.942397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:07.942419 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:07.943965 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.944362 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.944365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.944367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.944453 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:07.944720 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:07.944725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:07.944726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:07.944893 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:07.945038 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:07.945320 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 10:40:07.945440 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":37}
I20260924 10:40:08.899739 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:08.899311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.2:53375)
0924 10:40:08.899558 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:08.899645 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
W20260924 10:40:08.900111 9297 heartbeater.cc:644] Failed to heartbeat to 127.2.8.62:38611 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
I20260924 10:40:08.902767 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:08.902376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.3:52863)
0924 10:40:08.902616 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:08.902690 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
W20260924 10:40:08.903108 9433 heartbeater.cc:644] Failed to heartbeat to 127.2.8.62:38611 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
I20260924 10:40:08.908998 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:08.908566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.1:56629)
0924 10:40:08.908773 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:08.908839 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
W20260924 10:40:08.909309 9160 heartbeater.cc:644] Failed to heartbeat to 127.2.8.62:38611 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
I20260924 10:40:09.413669 9501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee68668fd32841e2a0495725f909eee7)
I20260924 10:40:09.413933 9501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } }
I20260924 10:40:09.414927 9501 leader_election.cc:288] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee68668fd32841e2a0495725f909eee7 (127.2.8.62:38611), 3192885cd0a94303817ce469f56c9ed2 (127.2.8.61:37391)
I20260924 10:40:09.415575 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.415097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.0.0.1:33186)
0924 10:40:09.415414 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.415489 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":69,"threads_started":1}
W20260924 10:40:09.415793 8967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
I20260924 10:40:09.416045 9505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee68668fd32841e2a0495725f909eee7)
I20260924 10:40:09.416262 9505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } }
I20260924 10:40:09.416641 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.416409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.0.0.1:33194)
0924 10:40:09.416537 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.416582 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 10:40:09.416771 8967 leader_election.cc:334] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ee68668fd32841e2a0495725f909eee7 (127.2.8.62:38611): Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
I20260924 10:40:09.417295 9505 leader_election.cc:288] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee68668fd32841e2a0495725f909eee7 (127.2.8.62:38611), 448a89e65a7c40c3b72df3b2ba71e68f (127.2.8.60:33075)
I20260924 10:40:09.417810 9508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.417398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.0.0.1:33196)
0924 10:40:09.417661 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.417726 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
W20260924 10:40:09.417994 8783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
I20260924 10:40:09.418807 9508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.418576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.0.0.1:33202)
0924 10:40:09.418691 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.418742 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 10:40:09.419003 8783 leader_election.cc:334] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ee68668fd32841e2a0495725f909eee7 (127.2.8.62:38611): Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
I20260924 10:40:09.420066 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.415231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.2.8.61:37391)
0924 10:40:09.415584 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:09.415591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:09.415613 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:40:09.415807 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:09.415813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:09.415866 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:09.415948 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:09.416354 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.417168 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.419004 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.419577 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.419656 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:09.419702 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:09.419915 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 10:40:09.419968 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":274,"thread_start_us":72,"threads_started":1}
I20260924 10:40:09.420109 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.415137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:37391 (local address 127.0.0.1:42326)
0924 10:40:09.415539 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.415554 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:09.415671 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:09.416077 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:09.416080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:09.416099 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:09.416198 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:09.416203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.417649 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.417652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:09.418897 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:09.418910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.419691 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.419696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:09.419787 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:09.419812 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:40:09.419895 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 10:40:09.419970 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":62,"threads_started":1}
I20260924 10:40:09.420636 8806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3192885cd0a94303817ce469f56c9ed2" is_pre_election: true
I20260924 10:40:09.420838 8806 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 448a89e65a7c40c3b72df3b2ba71e68f in term 1.
I20260924 10:40:09.421198 8964 leader_election.cc:302] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3192885cd0a94303817ce469f56c9ed2, 448a89e65a7c40c3b72df3b2ba71e68f; no voters: ee68668fd32841e2a0495725f909eee7
I20260924 10:40:09.421450 9501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 10:40:09.421522 9501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee68668fd32841e2a0495725f909eee7)
I20260924 10:40:09.421574 9501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:40:09.421765 9509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.417599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49162 (local address 127.2.8.60:33075)
0924 10:40:09.417820 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:09.417828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:09.418003 (+ 175us) server_negotiation.cc:407] Connection header received
0924 10:40:09.418057 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:09.418060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:09.418105 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:09.418195 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:09.418476 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.419205 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.420647 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.421209 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.421291 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:09.421377 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:09.421561 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 10:40:09.421659 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":106,"threads_started":1}
I20260924 10:40:09.422053 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.417473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:33075 (local address 127.0.0.1:49162)
0924 10:40:09.417831 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.417846 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:09.417985 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:09.418226 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:09.418230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:09.418254 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:09.418346 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:09.418352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.419363 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.419367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:09.420534 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:09.420543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.421319 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.421322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:09.421410 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:09.421432 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:40:09.421894 (+ 462us) client_negotiation.cc:240] Negotiation successful
0924 10:40:09.421972 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":60,"threads_started":1}
I20260924 10:40:09.422461 8982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3192885cd0a94303817ce469f56c9ed2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" is_pre_election: true
I20260924 10:40:09.422793 9501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } }
I20260924 10:40:09.423074 8982 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3192885cd0a94303817ce469f56c9ed2 in current term 2: Already voted for candidate 448a89e65a7c40c3b72df3b2ba71e68f in this term.
I20260924 10:40:09.423192 9501 leader_election.cc:288] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [CANDIDATE]: Term 2 election: Requested vote from peers ee68668fd32841e2a0495725f909eee7 (127.2.8.62:38611), 3192885cd0a94303817ce469f56c9ed2 (127.2.8.61:37391)
I20260924 10:40:09.423466 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.423261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.0.0.1:33214)
0924 10:40:09.423349 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.423402 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 10:40:09.423503 8806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3192885cd0a94303817ce469f56c9ed2"
I20260924 10:40:09.423606 8806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:40:09.423632 8782 leader_election.cc:302] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3192885cd0a94303817ce469f56c9ed2; no voters: 448a89e65a7c40c3b72df3b2ba71e68f, ee68668fd32841e2a0495725f909eee7
I20260924 10:40:09.424485 8806 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 448a89e65a7c40c3b72df3b2ba71e68f in term 2.
I20260924 10:40:09.424594 9505 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:40:09.424751 8964 leader_election.cc:302] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3192885cd0a94303817ce469f56c9ed2, 448a89e65a7c40c3b72df3b2ba71e68f; no voters:
I20260924 10:40:09.424899 9501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:40:09.425110 9501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [term 2 LEADER]: Becoming Leader. State: Replica: 448a89e65a7c40c3b72df3b2ba71e68f, State: Running, Role: LEADER
I20260924 10:40:09.425343 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.425125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.0.0.1:33226)
0924 10:40:09.425242 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.425292 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 10:40:09.425381 9501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } }
W20260924 10:40:09.425483 8967 leader_election.cc:334] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ee68668fd32841e2a0495725f909eee7 (127.2.8.62:38611): Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
I20260924 10:40:09.426783 9511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 448a89e65a7c40c3b72df3b2ba71e68f. Latest consensus state: current_term: 2 leader_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } } }
I20260924 10:40:09.426899 9511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:09.427178 9513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:40:09.429353 9513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:40:09.429739 9513 catalog_manager.cc:1259] Loaded cluster ID: 11926dce6cf44ca1acfbebd864657b29
I20260924 10:40:09.429809 9513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:40:09.430331 9513 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:40:09.430708 9513 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: Loaded TSK: 0
I20260924 10:40:09.431339 9513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:40:09.459746 8980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:40:09.462976 8980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:40:09.466185 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.465844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.0.0.1:33250)
0924 10:40:09.466023 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.466091 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 10:40:09.466192 8806 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [term 2 FOLLOWER]: Refusing update from remote peer 448a89e65a7c40c3b72df3b2ba71e68f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 10:40:09.466720 9501 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 10:40:09.466923 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.466678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.0.0.1:33264)
0924 10:40:09.466786 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.466841 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:40:09.467082 8967 consensus_peers.cc:487] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f -> Peer ee68668fd32841e2a0495725f909eee7 (127.2.8.62:38611): Couldn't send request to peer ee68668fd32841e2a0495725f909eee7. Status: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 10:40:09.468297 9505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 448a89e65a7c40c3b72df3b2ba71e68f. Latest consensus state: current_term: 2 leader_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } } }
I20260924 10:40:09.468421 9505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:40:09.469676 9511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } } }
I20260924 10:40:09.469796 9511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:09.470311 9505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ee68668fd32841e2a0495725f909eee7" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38611 } } peers { permanent_uuid: "3192885cd0a94303817ce469f56c9ed2" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 37391 } attrs { promote: false } } peers { permanent_uuid: "448a89e65a7c40c3b72df3b2ba71e68f" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 33075 } attrs { promote: false } } }
I20260924 10:40:09.470434 9505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3192885cd0a94303817ce469f56c9ed2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:40:09.480656 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.476087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:43511 (local address 127.0.0.1:42088)
0924 10:40:09.476312 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.476321 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:09.476378 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:09.476904 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:09.476908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:09.476927 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:09.477059 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:09.477068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.478105 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.478108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:09.479301 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:09.479310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.480194 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.480198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:09.480307 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:09.480337 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:40:09.480432 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 10:40:09.480509 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260924 10:40:09.480997 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.476339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42088 (local address 127.2.8.3:43511)
0924 10:40:09.476620 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:09.476625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:09.476640 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:40:09.476694 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:09.476699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:09.476752 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:09.476887 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:09.477191 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.477987 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.479430 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.480077 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.480175 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:09.480240 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:09.480453 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 10:40:09.480585 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260924 10:40:09.481099 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.476494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:42063 (local address 127.0.0.1:56864)
0924 10:40:09.477109 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.477124 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:09.477210 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:09.477483 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:09.477486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:09.477506 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:09.477609 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:09.477615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.478553 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.478559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:09.479524 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:09.479533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.480579 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.480583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:09.480671 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:09.480702 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 10:40:09.480791 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 10:40:09.480878 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":22,"thread_start_us":87,"threads_started":1}
I20260924 10:40:09.481196 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.475875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:35103 (local address 127.0.0.1:56712)
0924 10:40:09.476068 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.476082 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:09.476154 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:09.477646 (+ 1492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:09.477649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:09.477664 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:09.477745 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:09.477750 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.478745 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.478748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:09.479825 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:09.479833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.480750 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.480754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:09.480855 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:09.480876 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:40:09.480924 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 10:40:09.481028 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260924 10:40:09.481477 9366 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3428fe106f42c782bc00ba1e79295d (DEFAULT_TABLE table=test-table [id=045443b80a5547258bd68cb9112222ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:09.481637 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.475989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.2.8.2:35103)
0924 10:40:09.477431 (+ 1442us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:09.477438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:09.477453 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:40:09.477499 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:09.477502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:09.477546 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:09.477625 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:09.477859 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.478623 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.479966 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.480603 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.481334 (+ 731us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:09.481394 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:09.481442 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 10:40:09.481514 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1341,"thread_start_us":71,"threads_started":1}
I20260924 10:40:09.481644 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.476604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56864 (local address 127.2.8.1:42063)
0924 10:40:09.476827 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:09.476831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:09.477257 (+ 426us) server_negotiation.cc:407] Connection header received
0924 10:40:09.477316 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:09.477319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:09.477364 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:09.477452 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:09.477746 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.478437 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.479661 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.480359 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.480462 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:09.480523 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:09.481388 (+ 865us) server_negotiation.cc:299] Negotiation successful
0924 10:40:09.481467 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":62,"threads_started":1}
I20260924 10:40:09.481801 9366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3428fe106f42c782bc00ba1e79295d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:09.484107 9524 tablet_bootstrap.cc:492] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a: Bootstrap starting.
I20260924 10:40:09.484616 9093 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3428fe106f42c782bc00ba1e79295d (DEFAULT_TABLE table=test-table [id=045443b80a5547258bd68cb9112222ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:09.484663 9230 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3428fe106f42c782bc00ba1e79295d (DEFAULT_TABLE table=test-table [id=045443b80a5547258bd68cb9112222ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:09.484866 9230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3428fe106f42c782bc00ba1e79295d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:09.484910 9093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3428fe106f42c782bc00ba1e79295d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:09.485176 9524 tablet_bootstrap.cc:654] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:09.486575 9524 tablet_bootstrap.cc:492] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a: No bootstrap required, opened a new log
I20260924 10:40:09.486677 9524 ts_tablet_manager.cc:1397] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:40:09.487141 9524 raft_consensus.cc:348] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } } peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } }
I20260924 10:40:09.487284 9524 raft_consensus.cc:374] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:09.487335 9524 raft_consensus.cc:729] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2b0cf23c7c40fd8509b8438ea59d1a, State: Initialized, Role: FOLLOWER
I20260924 10:40:09.487232 9526 tablet_bootstrap.cc:492] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033: Bootstrap starting.
I20260924 10:40:09.487462 9524 consensus_queue.cc:260] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } } peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } }
I20260924 10:40:09.487669 9527 tablet_bootstrap.cc:492] T 5f3428fe106f42c782bc00ba1e79295d P 0bd5a2a65f554fdc95fe660cf67364ff: Bootstrap starting.
I20260924 10:40:09.487841 9524 ts_tablet_manager.cc:1428] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 10:40:09.488641 9527 tablet_bootstrap.cc:654] T 5f3428fe106f42c782bc00ba1e79295d P 0bd5a2a65f554fdc95fe660cf67364ff: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:09.489234 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.488807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.3:60199)
0924 10:40:09.489074 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.489137 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":125,"threads_started":1}
I20260924 10:40:09.490238 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.489750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.3:47383)
0924 10:40:09.490098 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.490156 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":95,"threads_started":1}
I20260924 10:40:09.490510 9527 tablet_bootstrap.cc:492] T 5f3428fe106f42c782bc00ba1e79295d P 0bd5a2a65f554fdc95fe660cf67364ff: No bootstrap required, opened a new log
I20260924 10:40:09.490626 9527 ts_tablet_manager.cc:1397] T 5f3428fe106f42c782bc00ba1e79295d P 0bd5a2a65f554fdc95fe660cf67364ff: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 10:40:09.490942 9432 heartbeater.cc:503] Master 127.2.8.60:33075 was elected leader, sending a full tablet report...
I20260924 10:40:09.491084 9527 raft_consensus.cc:348] T 5f3428fe106f42c782bc00ba1e79295d P 0bd5a2a65f554fdc95fe660cf67364ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } } peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } }
I20260924 10:40:09.491174 9526 tablet_bootstrap.cc:654] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:09.491215 9527 raft_consensus.cc:374] T 5f3428fe106f42c782bc00ba1e79295d P 0bd5a2a65f554fdc95fe660cf67364ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:09.491254 9527 raft_consensus.cc:729] T 5f3428fe106f42c782bc00ba1e79295d P 0bd5a2a65f554fdc95fe660cf67364ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bd5a2a65f554fdc95fe660cf67364ff, State: Initialized, Role: FOLLOWER
I20260924 10:40:09.491453 9527 consensus_queue.cc:260] T 5f3428fe106f42c782bc00ba1e79295d P 0bd5a2a65f554fdc95fe660cf67364ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } } peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } }
I20260924 10:40:09.491806 9527 ts_tablet_manager.cc:1428] T 5f3428fe106f42c782bc00ba1e79295d P 0bd5a2a65f554fdc95fe660cf67364ff: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:40:09.492801 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.492352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.1:56807)
0924 10:40:09.492609 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.492665 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":98,"threads_started":1}
I20260924 10:40:09.493572 9159 heartbeater.cc:503] Master 127.2.8.60:33075 was elected leader, sending a full tablet report...
I20260924 10:40:09.493708 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.493260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.1:33081)
0924 10:40:09.493529 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.493591 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":97,"threads_started":1}
I20260924 10:40:09.494076 9526 tablet_bootstrap.cc:492] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033: No bootstrap required, opened a new log
I20260924 10:40:09.494181 9526 ts_tablet_manager.cc:1397] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260924 10:40:09.494534 9526 raft_consensus.cc:348] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } } peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } }
I20260924 10:40:09.494611 9526 raft_consensus.cc:374] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:09.494632 9526 raft_consensus.cc:729] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b40ad03a8884554a439d5ac6838d033, State: Initialized, Role: FOLLOWER
I20260924 10:40:09.494761 9526 consensus_queue.cc:260] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } } peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } }
I20260924 10:40:09.495117 9526 ts_tablet_manager.cc:1428] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:40:09.496043 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.495678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.2:45511)
0924 10:40:09.495893 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.495956 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":103,"threads_started":1}
I20260924 10:40:09.496872 9296 heartbeater.cc:503] Master 127.2.8.60:33075 was elected leader, sending a full tablet report...
I20260924 10:40:09.497037 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.496630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.2:46507)
0924 10:40:09.496894 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.496944 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260924 10:40:09.572662 9528 raft_consensus.cc:482] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:40:09.572835 9528 raft_consensus.cc:504] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } } peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } }
I20260924 10:40:09.573312 9528 leader_election.cc:288] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b40ad03a8884554a439d5ac6838d033 (127.2.8.2:35103), 0bd5a2a65f554fdc95fe660cf67364ff (127.2.8.1:42063)
I20260924 10:40:09.573733 9113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3428fe106f42c782bc00ba1e79295d" candidate_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" is_pre_election: true
I20260924 10:40:09.573791 9250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3428fe106f42c782bc00ba1e79295d" candidate_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b40ad03a8884554a439d5ac6838d033" is_pre_election: true
I20260924 10:40:09.573938 9113 raft_consensus.cc:2454] T 5f3428fe106f42c782bc00ba1e79295d P 0bd5a2a65f554fdc95fe660cf67364ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2b0cf23c7c40fd8509b8438ea59d1a in term 0.
I20260924 10:40:09.574033 9250 raft_consensus.cc:2454] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad2b0cf23c7c40fd8509b8438ea59d1a in term 0.
I20260924 10:40:09.574342 9322 leader_election.cc:302] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bd5a2a65f554fdc95fe660cf67364ff, ad2b0cf23c7c40fd8509b8438ea59d1a; no voters:
I20260924 10:40:09.574509 9528 raft_consensus.cc:2781] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:40:09.574564 9528 raft_consensus.cc:482] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:40:09.574612 9528 raft_consensus.cc:3037] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:09.575709 9528 raft_consensus.cc:504] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } } peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } }
I20260924 10:40:09.576057 9528 leader_election.cc:288] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [CANDIDATE]: Term 1 election: Requested vote from peers 6b40ad03a8884554a439d5ac6838d033 (127.2.8.2:35103), 0bd5a2a65f554fdc95fe660cf67364ff (127.2.8.1:42063)
I20260924 10:40:09.576438 9113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3428fe106f42c782bc00ba1e79295d" candidate_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bd5a2a65f554fdc95fe660cf67364ff"
I20260924 10:40:09.576548 9113 raft_consensus.cc:3037] T 5f3428fe106f42c782bc00ba1e79295d P 0bd5a2a65f554fdc95fe660cf67364ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:09.576735 9250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3428fe106f42c782bc00ba1e79295d" candidate_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b40ad03a8884554a439d5ac6838d033"
I20260924 10:40:09.576862 9250 raft_consensus.cc:3037] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:09.577754 9113 raft_consensus.cc:2454] T 5f3428fe106f42c782bc00ba1e79295d P 0bd5a2a65f554fdc95fe660cf67364ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2b0cf23c7c40fd8509b8438ea59d1a in term 1.
I20260924 10:40:09.578038 9250 raft_consensus.cc:2454] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad2b0cf23c7c40fd8509b8438ea59d1a in term 1.
I20260924 10:40:09.578047 9322 leader_election.cc:302] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bd5a2a65f554fdc95fe660cf67364ff, ad2b0cf23c7c40fd8509b8438ea59d1a; no voters:
I20260924 10:40:09.578251 9528 raft_consensus.cc:2781] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:09.578523 9528 raft_consensus.cc:686] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [term 1 LEADER]: Becoming Leader. State: Replica: ad2b0cf23c7c40fd8509b8438ea59d1a, State: Running, Role: LEADER
I20260924 10:40:09.578670 9528 consensus_queue.cc:237] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } } peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } }
I20260924 10:40:09.579380 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.579058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.3:45523)
0924 10:40:09.579194 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.579283 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 10:40:09.580351 8979 catalog_manager.cc:5495] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad2b0cf23c7c40fd8509b8438ea59d1a (127.2.8.3). New cstate: current_term: 1 leader_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad2b0cf23c7c40fd8509b8438ea59d1a" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 43511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 } health_report { overall_health: UNKNOWN } } }
I20260924 10:40:09.605517 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.602364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42098 (local address 127.2.8.3:43511)
0924 10:40:09.602519 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:09.602526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:09.602557 (+ 31us) server_negotiation.cc:407] Connection header received
0924 10:40:09.602613 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:09.602615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:09.602658 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:09.602742 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:40:09.603016 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.603534 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.604401 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:09.604624 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:09.604736 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:09.605016 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:40:09.605098 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:40:09.605182 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:09.605306 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 10:40:09.605362 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 10:40:09.607614 9113 raft_consensus.cc:1261] T 5f3428fe106f42c782bc00ba1e79295d P 0bd5a2a65f554fdc95fe660cf67364ff [term 1 FOLLOWER]: Refusing update from remote peer ad2b0cf23c7c40fd8509b8438ea59d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:09.607632 9250 raft_consensus.cc:1261] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033 [term 1 FOLLOWER]: Refusing update from remote peer ad2b0cf23c7c40fd8509b8438ea59d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:09.608085 9528 consensus_queue.cc:1035] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bd5a2a65f554fdc95fe660cf67364ff" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 42063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:09.608280 9538 consensus_queue.cc:1035] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b40ad03a8884554a439d5ac6838d033" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:09.609975 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.609741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.1:44551)
0924 10:40:09.609831 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.609894 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 10:40:09.610208 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.609938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.2:52733)
0924 10:40:09.610079 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.610133 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 10:40:09.613124 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.612787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.3:56935)
0924 10:40:09.612988 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.613055 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260924 10:40:09.614957 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.612637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.1:52219)
0924 10:40:09.612888 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.612956 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260924 10:40:09.615850 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.614696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.2:55709)
0924 10:40:09.615728 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.615784 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":961}
I20260924 10:40:09.616629 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.614440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38611 (local address 127.2.8.3:51373)
0924 10:40:09.616509 (+ 2069us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.616562 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:38611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1996}
I20260924 10:40:09.620507 8978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:40:09.620671 8978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:40:09.624320 8978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 448a89e65a7c40c3b72df3b2ba71e68f: Sending DeleteTablet for 3 replicas of tablet 5f3428fe106f42c782bc00ba1e79295d
I20260924 10:40:09.624975 9230 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f3428fe106f42c782bc00ba1e79295d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:09 UTC) from {username='oryx', principal='oryx/127.2.8.60@KRBTEST.COM'} at 127.0.0.1:56712
I20260924 10:40:09.625344 9093 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f3428fe106f42c782bc00ba1e79295d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:09 UTC) from {username='oryx', principal='oryx/127.2.8.60@KRBTEST.COM'} at 127.0.0.1:56864
I20260924 10:40:09.625468 9366 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f3428fe106f42c782bc00ba1e79295d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:09 UTC) from {username='oryx', principal='oryx/127.2.8.60@KRBTEST.COM'} at 127.0.0.1:42088
I20260924 10:40:09.625789 9551 tablet_replica.cc:330] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a: stopping tablet replica
I20260924 10:40:09.625890 9550 tablet_replica.cc:330] T 5f3428fe106f42c782bc00ba1e79295d P 0bd5a2a65f554fdc95fe660cf67364ff: stopping tablet replica
I20260924 10:40:09.626001 9551 raft_consensus.cc:2229] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:40:09.626132 9550 raft_consensus.cc:2229] T 5f3428fe106f42c782bc00ba1e79295d P 0bd5a2a65f554fdc95fe660cf67364ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:40:09.626312 9550 raft_consensus.cc:2258] T 5f3428fe106f42c782bc00ba1e79295d P 0bd5a2a65f554fdc95fe660cf67364ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:09.626334 9551 raft_consensus.cc:2258] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:09.626736 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 8940
I20260924 10:40:09.627061 9551 ts_tablet_manager.cc:1905] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:09.629416 9549 tablet_replica.cc:330] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033: stopping tablet replica
I20260924 10:40:09.629556 9551 ts_tablet_manager.cc:1918] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:40:09.629562 9549 raft_consensus.cc:2229] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:40:09.629634 9551 log.cc:1196] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/5f3428fe106f42c782bc00ba1e79295d
I20260924 10:40:09.629712 9549 raft_consensus.cc:2258] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:09.630162 9551 ts_tablet_manager.cc:1939] T 5f3428fe106f42c782bc00ba1e79295d P ad2b0cf23c7c40fd8509b8438ea59d1a: Deleting consensus metadata
I20260924 10:40:09.630330 9549 ts_tablet_manager.cc:1905] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:09.630861 8967 catalog_manager.cc:4841] TS ad2b0cf23c7c40fd8509b8438ea59d1a (127.2.8.3:43511): tablet 5f3428fe106f42c782bc00ba1e79295d (table test-table [id=045443b80a5547258bd68cb9112222ff]) successfully deleted
I20260924 10:40:09.632377 9549 ts_tablet_manager.cc:1918] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:40:09.632471 9549 log.cc:1196] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/5f3428fe106f42c782bc00ba1e79295d
I20260924 10:40:09.632707 9549 ts_tablet_manager.cc:1939] T 5f3428fe106f42c782bc00ba1e79295d P 6b40ad03a8884554a439d5ac6838d033: Deleting consensus metadata
I20260924 10:40:09.633715 8966 catalog_manager.cc:4841] TS 6b40ad03a8884554a439d5ac6838d033 (127.2.8.2:35103): tablet 5f3428fe106f42c782bc00ba1e79295d (table test-table [id=045443b80a5547258bd68cb9112222ff]) successfully deleted
I20260924 10:40:09.639237 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 9167
I20260924 10:40:09.639509 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:09.639249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:42063 (local address 127.0.0.1:56876)
0924 10:40:09.639379 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:09.639431 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:42063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:40:09.639945 8967 catalog_manager.cc:4584] TS 0bd5a2a65f554fdc95fe660cf67364ff (127.2.8.1:42063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f3428fe106f42c782bc00ba1e79295d: Network error: Client connection negotiation failed: client connection to 127.2.8.1:42063: connect: Connection refused (error 111)
I20260924 10:40:09.645798 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 9303
I20260924 10:40:09.652437 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 8659
I20260924 10:40:09.658133 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 8768
2026-09-24T10:40:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5548 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:40:09 dist-test-slave-j38h krb5kdc[9554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:40:09 dist-test-slave-j38h krb5kdc[9554](info): set up 2 sockets
Sep 24 10:40:09 dist-test-slave-j38h krb5kdc[9554](info): commencing operation
krb5kdc: starting...
W20260924 10:40:10.781642 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:40:10 dist-test-slave-j38h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:40:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:40:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.62.keytab.
Entry for principal oryx/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.62.keytab.
I20260924 10:40:10.902426 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:39019
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.62.keytab
--principal=oryx/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:39019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:11.010136 9567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:11.010425 9567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:11.010468 9567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:11.014628 9567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:40:11.014698 9567 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 10:40:11.014716 9567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:11.014735 9567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:11.014751 9567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:40:11.014766 9567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:40:11.020200 9567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:39019
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.62.keytab
--principal=oryx/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:39019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:11.021379 9567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:11.022409 9567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:11.030918 9572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:11.031169 9567 server_base.cc:1034] running on GCE node
W20260924 10:40:11.030918 9575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:11.030917 9573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:11.031965 9567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:11.033109 9567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:11.034271 9567 hybrid_clock.cc:648] HybridClock initialized: now 1790246411034258 us; error 32 us; skew 500 ppm
Sep 24 10:40:11 dist-test-slave-j38h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246411, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:11.037096 9567 init.cc:377] Logged in from keytab as oryx/127.2.8.62@KRBTEST.COM (short username oryx)
I20260924 10:40:11.037997 9567 webserver.cc:460] Webserver started at http://127.2.8.62:38485/ using document root <none> and password file <none>
I20260924 10:40:11.038529 9567 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:11.038589 9567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:11.038822 9567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:11.040796 9567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "0afdc6214a9a46cca4cbafacfb870c66"
format_stamp: "Formatted at 2026-09-24 10:40:11 on dist-test-slave-j38h"
server_key: "827165fe2c288d3bdd66abd4135f658d"
server_key_iv: "7e3dd677fbb0228e3e1786c8fba89440"
server_key_version: "encryptionkey@0"
I20260924 10:40:11.041420 9567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "0afdc6214a9a46cca4cbafacfb870c66"
format_stamp: "Formatted at 2026-09-24 10:40:11 on dist-test-slave-j38h"
server_key: "827165fe2c288d3bdd66abd4135f658d"
server_key_iv: "7e3dd677fbb0228e3e1786c8fba89440"
server_key_version: "encryptionkey@0"
I20260924 10:40:11.044860 9567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:40:11.047149 9582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:11.048344 9567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:40:11.048482 9567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "0afdc6214a9a46cca4cbafacfb870c66"
format_stamp: "Formatted at 2026-09-24 10:40:11 on dist-test-slave-j38h"
server_key: "827165fe2c288d3bdd66abd4135f658d"
server_key_iv: "7e3dd677fbb0228e3e1786c8fba89440"
server_key_version: "encryptionkey@0"
I20260924 10:40:11.048604 9567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:11.062704 9567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:11.065996 9567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:11.066205 9567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:11.073973 9567 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:39019
I20260924 10:40:11.073977 9644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:39019 every 8 connection(s)
I20260924 10:40:11.074904 9567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:40:11.077628 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:11.078200 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 9567
I20260924 10:40:11.078336 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:40:11.078625 2080 external_mini_cluster.cc:1428] Setting key a85b4fd40602a711f74c81fe39754fa7
I20260924 10:40:11.084527 9645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66: Bootstrap starting.
I20260924 10:40:11.086779 9645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66: Neither blocks nor log segments found. Creating new log.
Sep 24 10:40:11 dist-test-slave-j38h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM
I20260924 10:40:11.087663 9645 log.cc:824] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:11.089570 9645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66: No bootstrap required, opened a new log
I20260924 10:40:11.093046 9645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afdc6214a9a46cca4cbafacfb870c66" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 39019 } }
I20260924 10:40:11.093255 9645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:11.093214 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.079885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.2.8.62:39019)
0924 10:40:11.080287 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:11.080301 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:11.080335 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:40:11.080948 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:11.080966 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:11.081361 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:11.081732 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:11.082558 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.083519 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.084378 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.084899 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.085086 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.087851 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:11.087856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:11.087868 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:11.087894 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:11.090149 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:11.090657 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:11.090662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:11.090667 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:11.090814 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:11.091116 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:11.091119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:11.091120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:11.091588 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:11.091743 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:11.092042 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 10:40:11.092420 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":141,"threads_started":1}
I20260924 10:40:11.093314 9645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afdc6214a9a46cca4cbafacfb870c66, State: Initialized, Role: FOLLOWER
I20260924 10:40:11.093775 9645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afdc6214a9a46cca4cbafacfb870c66" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 39019 } }
I20260924 10:40:11.093928 9645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:40:11.094022 9645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:40:11.094141 9645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:11.095249 9645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afdc6214a9a46cca4cbafacfb870c66" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 39019 } }
I20260924 10:40:11.095597 9645 leader_election.cc:302] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0afdc6214a9a46cca4cbafacfb870c66; no voters:
I20260924 10:40:11.095939 9645 leader_election.cc:288] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:40:11.096014 9650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:11.096354 9650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [term 1 LEADER]: Becoming Leader. State: Replica: 0afdc6214a9a46cca4cbafacfb870c66, State: Running, Role: LEADER
I20260924 10:40:11.096647 9650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afdc6214a9a46cca4cbafacfb870c66" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 39019 } }
I20260924 10:40:11.097208 9645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:40:11.098376 9652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0afdc6214a9a46cca4cbafacfb870c66. Latest consensus state: current_term: 1 leader_uuid: "0afdc6214a9a46cca4cbafacfb870c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afdc6214a9a46cca4cbafacfb870c66" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 39019 } } }
I20260924 10:40:11.098505 9652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:11.098837 9651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0afdc6214a9a46cca4cbafacfb870c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afdc6214a9a46cca4cbafacfb870c66" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 39019 } } }
I20260924 10:40:11.098934 9659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:40:11.098943 9651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:11.102105 9659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:40:11.107686 9659 catalog_manager.cc:1347] Generated new cluster ID: 376a6ef6b11440818aec807a7a504214
I20260924 10:40:11.107772 9659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:40:11.145181 9659 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:40:11.146231 9659 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:40:11.160215 9659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66: Generated new TSK 0
I20260924 10:40:11.161135 9659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.1.keytab.
Entry for principal oryx/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.1.keytab.
I20260924 10:40:11.205948 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:39019
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.1.keytab
--principal=oryx/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:11.331238 9671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:11.331504 9671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:11.331580 9671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:11.336100 9671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:11.336216 9671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:11.336308 9671 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:40:11.336400 9671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:40:11.342190 9671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.1.keytab
--principal=oryx/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.8.62:39019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:11.343276 9671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:11.344316 9671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:11.353874 9677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:11.353890 9676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:11.353895 9679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:11.354405 9671 server_base.cc:1034] running on GCE node
I20260924 10:40:11.354732 9671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:11.355281 9671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:11.356459 9671 hybrid_clock.cc:648] HybridClock initialized: now 1790246411356436 us; error 35 us; skew 500 ppm
Sep 24 10:40:11 dist-test-slave-j38h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246411, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:11.359243 9671 init.cc:377] Logged in from keytab as oryx/127.2.8.1@KRBTEST.COM (short username oryx)
I20260924 10:40:11.360097 9671 webserver.cc:460] Webserver started at http://127.2.8.1:43003/ using document root <none> and password file <none>
I20260924 10:40:11.360647 9671 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:11.360710 9671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:11.360942 9671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:11.362946 9671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "6b7dc901b82a487994426957fd003853"
format_stamp: "Formatted at 2026-09-24 10:40:11 on dist-test-slave-j38h"
server_key: "49aca8ea18f370eacb76b6e3938cc474"
server_key_iv: "4414b3c8ba8be397fcd5c5beb15eff57"
server_key_version: "encryptionkey@0"
I20260924 10:40:11.363485 9671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "6b7dc901b82a487994426957fd003853"
format_stamp: "Formatted at 2026-09-24 10:40:11 on dist-test-slave-j38h"
server_key: "49aca8ea18f370eacb76b6e3938cc474"
server_key_iv: "4414b3c8ba8be397fcd5c5beb15eff57"
server_key_version: "encryptionkey@0"
I20260924 10:40:11.366995 9671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 10:40:11.369270 9686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:11.370461 9671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:40:11.370584 9671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "6b7dc901b82a487994426957fd003853"
format_stamp: "Formatted at 2026-09-24 10:40:11 on dist-test-slave-j38h"
server_key: "49aca8ea18f370eacb76b6e3938cc474"
server_key_iv: "4414b3c8ba8be397fcd5c5beb15eff57"
server_key_version: "encryptionkey@0"
I20260924 10:40:11.370677 9671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:11.401628 9671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:11.404654 9671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:11.404862 9671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:11.406311 9671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:11.406358 9671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:11.406427 9671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:11.406462 9671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:40:11 dist-test-slave-j38h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246411, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM
I20260924 10:40:11.417546 9671 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:45945
I20260924 10:40:11.417564 9801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:45945 every 8 connection(s)
I20260924 10:40:11.418411 9671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:40:11.420758 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.407257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:37271 (local address 127.2.8.62:39019)
0924 10:40:11.407382 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:11.407386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:11.408412 (+ 1026us) server_negotiation.cc:407] Connection header received
0924 10:40:11.409621 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:11.409624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:11.409684 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:11.409777 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:11.411394 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.412098 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.413928 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.414169 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.414291 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.416878 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:11.416885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:11.416892 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:11.416924 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:11.418644 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:11.419241 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:11.419245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:11.419247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:11.419340 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:11.419677 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:11.419682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:11.419683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:11.419868 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:11.419977 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:11.420464 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 10:40:11.420588 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:40:11.421550 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.407809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:39019 (local address 127.2.8.1:37271)
0924 10:40:11.408266 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:11.408307 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:11.409346 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:11.409993 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:11.410006 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:11.410480 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:11.411207 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.411232 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.412224 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.412228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.413775 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.413784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.414295 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.414352 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.414562 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.415165 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:11.415181 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:11.416705 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:11.418764 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:11.418769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:11.418783 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:11.419149 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:11.419434 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:11.419439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:11.419440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:11.419580 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:11.419979 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:11.419988 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:11.420228 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 10:40:11.420412 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 10:40:11.420710 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":123,"threads_started":1}
I20260924 10:40:11.422791 9802 heartbeater.cc:347] Connected to a master server at 127.2.8.62:39019
I20260924 10:40:11.422847 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 9671
I20260924 10:40:11.422993 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:40:11.423285 2080 external_mini_cluster.cc:1428] Setting key 638682c032d95ac0e15c9cc9b9a6ee5e
I20260924 10:40:11.424966 9802 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:11.425484 9802 heartbeater.cc:511] Master 127.2.8.62:39019 requested a full tablet report, sending...
I20260924 10:40:11.427187 9599 ts_manager.cc:194] Registered new tserver with Master: 6b7dc901b82a487994426957fd003853 (127.2.8.1:45945)
I20260924 10:40:11.428785 9599 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:37271
WARNING: no policy specified for oryx/127.2.8.2@KRBTEST.COM; defaulting to no policy
I20260924 10:40:11.433257 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.426210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:42551 (local address 127.2.8.62:39019)
0924 10:40:11.426321 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:11.426325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:11.426337 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:40:11.426368 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:11.426374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:11.426412 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:11.426510 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:11.426916 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.427502 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.428345 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.428617 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.428713 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.429875 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:11.429877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:11.429879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:11.429898 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:11.431564 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:11.432061 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:11.432063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:11.432064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:11.432134 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:11.432444 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:11.432446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:11.432447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:11.432593 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:11.432664 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:11.433020 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 10:40:11.433117 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:40:11.433339 9808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.425094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:39019 (local address 127.2.8.1:42551)
0924 10:40:11.425393 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:11.425410 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:11.425496 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:11.426525 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:11.426528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:11.426713 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:11.426800 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.426805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.427619 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.427624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.428232 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.428240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.428702 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.428718 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.428807 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.429323 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:11.429334 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:11.429660 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:11.431688 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:11.431693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:11.431695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:11.431943 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:11.432233 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:11.432239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:11.432244 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:11.432349 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:11.432693 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:11.432699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:11.432801 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 10:40:11.432920 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 10:40:11.433103 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.2.keytab.
Entry for principal oryx/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.2.keytab.
I20260924 10:40:11.451586 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:39019
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.2.keytab
--principal=oryx/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:11.559592 9811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:11.559854 9811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:11.559898 9811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:11.564134 9811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:11.564198 9811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:11.564263 9811 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:40:11.564329 9811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:40:11.569751 9811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.2.keytab
--principal=oryx/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.8.62:39019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:11.570814 9811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:11.571838 9811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:11.581449 9816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:11.581526 9811 server_base.cc:1034] running on GCE node
W20260924 10:40:11.581455 9817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:11.581427 9819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:11.582039 9811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:11.582651 9811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:11.583839 9811 hybrid_clock.cc:648] HybridClock initialized: now 1790246411583818 us; error 32 us; skew 500 ppm
Sep 24 10:40:11 dist-test-slave-j38h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246411, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:11.586620 9811 init.cc:377] Logged in from keytab as oryx/127.2.8.2@KRBTEST.COM (short username oryx)
I20260924 10:40:11.587466 9811 webserver.cc:460] Webserver started at http://127.2.8.2:45397/ using document root <none> and password file <none>
I20260924 10:40:11.588028 9811 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:11.588119 9811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:11.588351 9811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:11.590499 9811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "0a3be87ca68e428b99d5a6ed98b83c66"
format_stamp: "Formatted at 2026-09-24 10:40:11 on dist-test-slave-j38h"
server_key: "c4444077a6bfcb51dd4a4213dd627dcc"
server_key_iv: "6dca0eee530dbdf94a4859fd94214277"
server_key_version: "encryptionkey@0"
I20260924 10:40:11.591065 9811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "0a3be87ca68e428b99d5a6ed98b83c66"
format_stamp: "Formatted at 2026-09-24 10:40:11 on dist-test-slave-j38h"
server_key: "c4444077a6bfcb51dd4a4213dd627dcc"
server_key_iv: "6dca0eee530dbdf94a4859fd94214277"
server_key_version: "encryptionkey@0"
I20260924 10:40:11.594652 9811 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:40:11.597113 9826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:11.598465 9811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:40:11.598582 9811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "0a3be87ca68e428b99d5a6ed98b83c66"
format_stamp: "Formatted at 2026-09-24 10:40:11 on dist-test-slave-j38h"
server_key: "c4444077a6bfcb51dd4a4213dd627dcc"
server_key_iv: "6dca0eee530dbdf94a4859fd94214277"
server_key_version: "encryptionkey@0"
I20260924 10:40:11.598716 9811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:11.614326 9811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:11.617569 9811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:11.617771 9811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:11.619153 9811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:11.619231 9811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:11.619304 9811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:11.619341 9811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:40:11 dist-test-slave-j38h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246411, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM
I20260924 10:40:11.631341 9811 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:40615
I20260924 10:40:11.631390 9941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:40615 every 8 connection(s)
I20260924 10:40:11.632282 9811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:40:11.634605 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.620425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:48733 (local address 127.2.8.62:39019)
0924 10:40:11.620567 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:11.620571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:11.621495 (+ 924us) server_negotiation.cc:407] Connection header received
0924 10:40:11.622743 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:11.622746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:11.622806 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:11.622905 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:11.624600 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.625387 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.627268 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.627680 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.627778 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.631074 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:11.631081 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:11.631087 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:11.631123 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:11.632785 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:11.633336 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:11.633339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:11.633340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:11.633419 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:11.633689 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:11.633691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:11.633692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:11.633862 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:11.633936 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:11.634363 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 10:40:11.634464 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:40:11.635429 9835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.620865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:39019 (local address 127.2.8.2:48733)
0924 10:40:11.621337 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:11.621381 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:11.622492 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:11.623133 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:11.623143 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:11.623634 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:11.624401 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.624423 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.625568 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.625572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.627108 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.627120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.627927 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.627978 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.628177 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.628882 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:11.628907 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:11.630822 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:11.632911 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:11.632917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:11.632931 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:11.633242 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:11.633494 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:11.633496 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:11.633497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:11.633607 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:11.633947 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:11.633952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:11.634163 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 10:40:11.634302 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 10:40:11.634576 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":120,"threads_started":1}
I20260924 10:40:11.636584 9942 heartbeater.cc:347] Connected to a master server at 127.2.8.62:39019
I20260924 10:40:11.636742 9942 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:11.637296 9942 heartbeater.cc:511] Master 127.2.8.62:39019 requested a full tablet report, sending...
I20260924 10:40:11.637620 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 9811
I20260924 10:40:11.637741 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:40:11.638077 2080 external_mini_cluster.cc:1428] Setting key ee6e6a5d8c95e17bf7606839f74857e6
I20260924 10:40:11.638366 9598 ts_manager.cc:194] Registered new tserver with Master: 0a3be87ca68e428b99d5a6ed98b83c66 (127.2.8.2:40615)
I20260924 10:40:11.639144 9598 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:48733
I20260924 10:40:11.647007 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.639403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:52369 (local address 127.2.8.62:39019)
0924 10:40:11.639530 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:11.639535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:11.639549 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:40:11.639584 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:11.639586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:11.639625 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:11.639708 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:11.640147 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.640963 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.641828 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.642210 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.642332 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.643726 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:11.643729 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:11.643731 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:11.643752 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:11.645310 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:11.645816 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:11.645819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:11.645820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:11.645899 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:11.646205 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:11.646207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:11.646208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:11.646410 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:11.646486 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:11.646790 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 10:40:11.646894 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:40:11.647142 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.638887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:39019 (local address 127.2.8.2:52369)
0924 10:40:11.639151 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:11.639164 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:11.639270 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:11.639742 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:11.639746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:11.639931 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:11.640034 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.640040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.641148 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.641152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.641715 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.641722 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.642338 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.642361 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.642486 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.643053 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:11.643073 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:11.643568 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:11.645426 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:11.645432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:11.645437 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:11.645716 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:11.645996 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:11.645999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:11.646002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:11.646107 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:11.646507 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:11.646512 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:11.646621 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 10:40:11.646745 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 10:40:11.646901 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for oryx/127.2.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.3.keytab.
Entry for principal oryx/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.3.keytab.
I20260924 10:40:11.668294 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:39019
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.3.keytab
--principal=oryx/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:11.778656 9951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:11.778916 9951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:11.778963 9951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:11.783036 9951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:11.783097 9951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:11.783136 9951 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 10:40:11.783206 9951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:40:11.788431 9951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.3.keytab
--principal=oryx/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.8.62:39019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:11.789533 9951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:11.790565 9951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:11.800437 9957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:11.800524 9951 server_base.cc:1034] running on GCE node
W20260924 10:40:11.800535 9959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:11.800463 9956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:11.801142 9951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:11.801800 9951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:11.802992 9951 hybrid_clock.cc:648] HybridClock initialized: now 1790246411802975 us; error 33 us; skew 500 ppm
Sep 24 10:40:11 dist-test-slave-j38h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246411, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:11.805843 9951 init.cc:377] Logged in from keytab as oryx/127.2.8.3@KRBTEST.COM (short username oryx)
I20260924 10:40:11.806763 9951 webserver.cc:460] Webserver started at http://127.2.8.3:46237/ using document root <none> and password file <none>
I20260924 10:40:11.807340 9951 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:11.807402 9951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:11.807650 9951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:11.809767 9951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "0dadee832c554cd9bb35d52d6d608406"
format_stamp: "Formatted at 2026-09-24 10:40:11 on dist-test-slave-j38h"
server_key: "dbf57c27621a2a3a2823578750ba45c2"
server_key_iv: "abbd8c73d5843051cebc13d6a339131f"
server_key_version: "encryptionkey@0"
I20260924 10:40:11.810313 9951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "0dadee832c554cd9bb35d52d6d608406"
format_stamp: "Formatted at 2026-09-24 10:40:11 on dist-test-slave-j38h"
server_key: "dbf57c27621a2a3a2823578750ba45c2"
server_key_iv: "abbd8c73d5843051cebc13d6a339131f"
server_key_version: "encryptionkey@0"
I20260924 10:40:11.814301 9951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 10:40:11.816751 9966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:11.818095 9951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:40:11.818212 9951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "0dadee832c554cd9bb35d52d6d608406"
format_stamp: "Formatted at 2026-09-24 10:40:11 on dist-test-slave-j38h"
server_key: "dbf57c27621a2a3a2823578750ba45c2"
server_key_iv: "abbd8c73d5843051cebc13d6a339131f"
server_key_version: "encryptionkey@0"
I20260924 10:40:11.818321 9951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:11.843343 9951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:11.846707 9951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:11.846863 9951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:11.848286 9951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:11.848367 9951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:11.848439 9951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:11.848495 9951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 10:40:11 dist-test-slave-j38h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246411, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM
I20260924 10:40:11.864634 9951 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:39475
I20260924 10:40:11.864732 10081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:39475 every 8 connection(s)
I20260924 10:40:11.866190 9951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:40:11.867710 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.849086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:58111 (local address 127.2.8.62:39019)
0924 10:40:11.849217 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:11.849220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:11.850255 (+ 1035us) server_negotiation.cc:407] Connection header received
0924 10:40:11.851412 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:11.851419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:11.851477 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:11.851534 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:11.853675 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.854692 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.856730 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.857327 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.857438 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.862761 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:11.862769 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:11.862775 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:11.862812 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:11.864676 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:11.865703 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:11.865707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:11.865709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:11.865801 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:11.866164 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:11.866167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:11.866169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:11.866434 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:11.866549 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:11.867327 (+ 778us) server_negotiation.cc:299] Negotiation successful
0924 10:40:11.867484 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:40:11.868433 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.849609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:39019 (local address 127.2.8.3:58111)
0924 10:40:11.850092 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:11.850135 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:11.851080 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:11.851951 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:11.851980 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:11.852818 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:11.853496 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.853515 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.854895 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.854899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.856299 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.856309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.857609 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.857682 (+ 73us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.857917 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.859413 (+ 1496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:11.859437 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:11.862441 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:11.865179 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:11.865189 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:11.865207 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:11.865584 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:11.865904 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:11.865908 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:11.865910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:11.866058 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:11.866556 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:11.866565 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:11.866862 (+ 297us) client_negotiation.cc:769] Sending connection context
0924 10:40:11.867061 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 10:40:11.867395 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":127,"threads_started":1}
I20260924 10:40:11.869820 10082 heartbeater.cc:347] Connected to a master server at 127.2.8.62:39019
I20260924 10:40:11.869961 10082 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:11.870656 10082 heartbeater.cc:511] Master 127.2.8.62:39019 requested a full tablet report, sending...
I20260924 10:40:11.871752 9599 ts_manager.cc:194] Registered new tserver with Master: 0dadee832c554cd9bb35d52d6d608406 (127.2.8.3:39475)
I20260924 10:40:11.872557 9599 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:58111
I20260924 10:40:11.875213 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 9951
I20260924 10:40:11.875377 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:40:11.875737 2080 external_mini_cluster.cc:1428] Setting key f1df560d4830001002097dad7a906fe8
I20260924 10:40:11.878211 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:40:11.882825 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.872603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:49909 (local address 127.2.8.62:39019)
0924 10:40:11.872706 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:11.872710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:11.873075 (+ 365us) server_negotiation.cc:407] Connection header received
0924 10:40:11.873227 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:11.873230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:11.873277 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:11.873354 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:11.873913 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.874828 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.877415 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.877734 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.877844 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.879176 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:11.879181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:11.879186 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:11.879215 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:11.880740 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:11.881355 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:11.881360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:11.881364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:11.881483 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:11.881831 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:11.881837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:11.881842 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:11.882094 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:11.882183 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:11.882552 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 10:40:11.882681 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:40:11.883203 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.872301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:39019 (local address 127.2.8.3:49909)
0924 10:40:11.872998 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:11.873014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:11.873122 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:11.873394 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:11.873396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:11.873631 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:11.873763 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.873774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.876199 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.876205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.877240 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.877255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.877848 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.877868 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.877975 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.878511 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:11.878530 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:11.879007 (+ 477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:11.880898 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:11.880902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:11.880904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:11.881212 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:11.881595 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:11.881598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:11.881599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:11.881698 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:11.882242 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:11.882248 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:11.882354 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 10:40:11.882453 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 10:40:11.882627 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":77,"threads_started":1}
Sep 24 10:40:11 dist-test-slave-j38h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246411, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 10:40:11 dist-test-slave-j38h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:11 dist-test-slave-j38h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:40:11.903440 9648 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:11.897131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.2.8.62:39019)
0924 10:40:11.897265 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:11.897268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:11.897281 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:40:11.897324 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:11.897327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:11.897397 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:11.897476 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:11.898167 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.898819 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.899756 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.899975 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.900078 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.903307 (+ 3229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40242: BlockingRecv error: recv got EOF from 127.0.0.1:40242 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 10:40:11 dist-test-slave-j38h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246411, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM
I20260924 10:40:11.913190 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.905137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.2.8.62:39019)
0924 10:40:11.905273 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:11.905277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:11.905469 (+ 192us) server_negotiation.cc:407] Connection header received
0924 10:40:11.905552 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:11.905554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:11.905593 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:11.905663 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:11.906194 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.906741 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.907502 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.907713 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.907790 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.910103 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:11.910106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:11.910109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:11.910154 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:11.911398 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:11.911862 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:11.911871 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:11.911873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:11.911956 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:11.912251 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:11.912255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:11.912256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:11.912472 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:11.912623 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:11.912869 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 10:40:11.913017 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:40:11.916005 9599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:40:11.919322 9599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:40:11.938553 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.932250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:40615 (local address 127.0.0.1:44094)
0924 10:40:11.932656 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:11.932672 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:11.932753 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:11.933700 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:11.933706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:11.933729 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:11.933825 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.933835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.935268 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.935271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.936674 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.936685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.937968 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.937972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.938093 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.938151 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 10:40:11.938241 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 10:40:11.938370 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":77,"thread_start_us":82,"threads_started":1}
I20260924 10:40:11.938930 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.932015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36218 (local address 127.2.8.3:39475)
0924 10:40:11.932465 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:11.932479 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:11.932507 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:40:11.932743 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:11.932750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:11.933041 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:11.933185 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:11.933754 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.935104 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.936832 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.937806 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.937906 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.938095 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:11.938669 (+ 574us) server_negotiation.cc:299] Negotiation successful
0924 10:40:11.938789 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":114,"threads_started":1}
I20260924 10:40:11.938596 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.931883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:39475 (local address 127.0.0.1:36218)
0924 10:40:11.932267 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:11.932328 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:11.932526 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:11.933414 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:11.933432 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:11.933456 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:11.933609 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.933618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.935266 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.935270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.936688 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.936700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.938191 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.938195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.938319 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.938339 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:40:11.938407 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 10:40:11.938479 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":80,"thread_start_us":87,"threads_started":1}
I20260924 10:40:11.939594 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.932711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.2.8.2:40615)
0924 10:40:11.933315 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:11.933326 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:11.933355 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:40:11.933414 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:11.933417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:11.933552 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:11.933715 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:11.933977 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.935160 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.936877 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.937806 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.938809 (+ 1003us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.938998 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:11.939058 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 10:40:11.939152 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":79,"threads_started":1}
I20260924 10:40:11.939994 10108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.932708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:45945 (local address 127.0.0.1:54274)
0924 10:40:11.933946 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:11.933962 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:11.934045 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:11.936085 (+ 2040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:11.936088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:11.936106 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:11.936193 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.936199 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.937432 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.937437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.938276 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:11.938286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.939455 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.939459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:11.939543 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.939565 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:40:11.939629 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 10:40:11.939852 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260924 10:40:11.940622 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:11.933145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54274 (local address 127.2.8.1:45945)
0924 10:40:11.935382 (+ 2237us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:11.935389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:11.935414 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:40:11.935493 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:11.935500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:11.935685 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:11.935838 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:11.936326 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.937323 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.938396 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:11.939350 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:11.939462 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:11.939601 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:11.939651 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 10:40:11.940491 (+ 840us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2123,"thread_start_us":64,"threads_started":1}
I20260924 10:40:11.942281 9875 tablet_service.cc:1467] Processing CreateTablet for tablet 65c16d2756f34f1482b88938a70be0d2 (DEFAULT_TABLE table=test-table [id=1b4d0034df35423eac79e04a6afa6d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:11.942417 10006 tablet_service.cc:1467] Processing CreateTablet for tablet 65c16d2756f34f1482b88938a70be0d2 (DEFAULT_TABLE table=test-table [id=1b4d0034df35423eac79e04a6afa6d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:11.943291 9875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65c16d2756f34f1482b88938a70be0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:11.943495 10006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65c16d2756f34f1482b88938a70be0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:11.943178 9736 tablet_service.cc:1467] Processing CreateTablet for tablet 65c16d2756f34f1482b88938a70be0d2 (DEFAULT_TABLE table=test-table [id=1b4d0034df35423eac79e04a6afa6d0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:11.944111 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65c16d2756f34f1482b88938a70be0d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:11.949846 10110 tablet_bootstrap.cc:492] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66: Bootstrap starting.
I20260924 10:40:11.951162 10111 tablet_bootstrap.cc:492] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406: Bootstrap starting.
I20260924 10:40:11.952741 10110 tablet_bootstrap.cc:654] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:11.953245 10111 tablet_bootstrap.cc:654] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:11.953313 10112 tablet_bootstrap.cc:492] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853: Bootstrap starting.
I20260924 10:40:11.953934 10110 log.cc:824] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:11.954010 10111 log.cc:824] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:11.955412 10112 tablet_bootstrap.cc:654] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:11.955884 10111 tablet_bootstrap.cc:492] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406: No bootstrap required, opened a new log
I20260924 10:40:11.955883 10110 tablet_bootstrap.cc:492] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66: No bootstrap required, opened a new log
I20260924 10:40:11.956071 10111 ts_tablet_manager.cc:1397] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:40:11.956071 10110 ts_tablet_manager.cc:1397] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:40:11.956231 10112 log.cc:824] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:11.957960 10112 tablet_bootstrap.cc:492] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853: No bootstrap required, opened a new log
I20260924 10:40:11.958175 10112 ts_tablet_manager.cc:1397] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:40:11.960064 10111 raft_consensus.cc:348] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } }
I20260924 10:40:11.960064 10110 raft_consensus.cc:348] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } }
I20260924 10:40:11.960336 10111 raft_consensus.cc:374] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:11.960336 10110 raft_consensus.cc:374] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:11.960393 10111 raft_consensus.cc:729] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dadee832c554cd9bb35d52d6d608406, State: Initialized, Role: FOLLOWER
I20260924 10:40:11.960393 10110 raft_consensus.cc:729] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a3be87ca68e428b99d5a6ed98b83c66, State: Initialized, Role: FOLLOWER
I20260924 10:40:11.960857 10111 consensus_queue.cc:260] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } }
I20260924 10:40:11.960857 10110 consensus_queue.cc:260] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } }
I20260924 10:40:11.961545 10082 heartbeater.cc:503] Master 127.2.8.62:39019 was elected leader, sending a full tablet report...
I20260924 10:40:11.961829 9942 heartbeater.cc:503] Master 127.2.8.62:39019 was elected leader, sending a full tablet report...
I20260924 10:40:11.961886 10110 ts_tablet_manager.cc:1428] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:40:11.962831 10112 raft_consensus.cc:348] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } }
I20260924 10:40:11.963120 10112 raft_consensus.cc:374] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:11.963181 10112 raft_consensus.cc:729] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7dc901b82a487994426957fd003853, State: Initialized, Role: FOLLOWER
I20260924 10:40:11.963187 10111 ts_tablet_manager.cc:1428] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260924 10:40:11.963673 10112 consensus_queue.cc:260] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } }
I20260924 10:40:11.964874 9802 heartbeater.cc:503] Master 127.2.8.62:39019 was elected leader, sending a full tablet report...
I20260924 10:40:11.964718 10112 ts_tablet_manager.cc:1428] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
W20260924 10:40:12.117805 10083 tablet.cc:2369] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:40:12.133502 9943 tablet.cc:2369] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:40:12.169775 9803 tablet.cc:2369] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:40:12.313139 10116 raft_consensus.cc:482] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:40:12.313413 10116 raft_consensus.cc:504] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } }
I20260924 10:40:12.314620 10116 leader_election.cc:288] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dadee832c554cd9bb35d52d6d608406 (127.2.8.3:39475), 6b7dc901b82a487994426957fd003853 (127.2.8.1:45945)
I20260924 10:40:12.319530 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.315373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:40607 (local address 127.2.8.1:45945)
0924 10:40:12.315500 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:12.315504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:12.315717 (+ 213us) server_negotiation.cc:407] Connection header received
0924 10:40:12.315786 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:12.315788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:12.315857 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:12.315935 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:12.316161 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.316849 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.318279 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.318869 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.318978 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.319040 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:12.319293 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 10:40:12.319360 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:40:12.319609 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.314836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:39475 (local address 127.2.8.2:60017)
0924 10:40:12.315135 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:12.315151 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:12.315293 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:12.315574 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:12.315577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:12.315598 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:12.315744 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.315754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.316798 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.316800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.318154 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.318165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.318994 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.318997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.319150 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.319182 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 10:40:12.319260 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 10:40:12.319375 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":79,"thread_start_us":100,"threads_started":1}
I20260924 10:40:12.319607 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.315236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:45945 (local address 127.2.8.2:40607)
0924 10:40:12.315610 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:12.315624 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:12.315713 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:12.315954 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:12.315957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:12.315973 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:12.316050 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.316054 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.316947 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.316949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.318156 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.318165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.318994 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.318997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.319146 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.319196 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 10:40:12.319260 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 10:40:12.319375 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":99,"thread_start_us":99,"threads_started":1}
I20260924 10:40:12.319995 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.314987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:60017 (local address 127.2.8.3:39475)
0924 10:40:12.315135 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:12.315141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:12.315258 (+ 117us) server_negotiation.cc:407] Connection header received
0924 10:40:12.315417 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:12.315420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:12.315486 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:12.315555 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:12.315859 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.316687 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.318279 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.318869 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.318978 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.319040 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:12.319771 (+ 731us) server_negotiation.cc:299] Negotiation successful
0924 10:40:12.319862 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:40:12.320537 9754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c16d2756f34f1482b88938a70be0d2" candidate_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7dc901b82a487994426957fd003853" is_pre_election: true
I20260924 10:40:12.320554 10033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c16d2756f34f1482b88938a70be0d2" candidate_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dadee832c554cd9bb35d52d6d608406" is_pre_election: true
I20260924 10:40:12.320837 10033 raft_consensus.cc:2454] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a3be87ca68e428b99d5a6ed98b83c66 in term 0.
I20260924 10:40:12.320848 9754 raft_consensus.cc:2454] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a3be87ca68e428b99d5a6ed98b83c66 in term 0.
I20260924 10:40:12.321326 9829 leader_election.cc:302] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a3be87ca68e428b99d5a6ed98b83c66, 0dadee832c554cd9bb35d52d6d608406; no voters:
I20260924 10:40:12.321597 10116 raft_consensus.cc:2781] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:40:12.321694 10116 raft_consensus.cc:482] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:40:12.321753 10116 raft_consensus.cc:3037] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:12.322850 10116 raft_consensus.cc:504] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } }
I20260924 10:40:12.323236 10116 leader_election.cc:288] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [CANDIDATE]: Term 1 election: Requested vote from peers 0dadee832c554cd9bb35d52d6d608406 (127.2.8.3:39475), 6b7dc901b82a487994426957fd003853 (127.2.8.1:45945)
I20260924 10:40:12.323549 10033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c16d2756f34f1482b88938a70be0d2" candidate_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dadee832c554cd9bb35d52d6d608406"
I20260924 10:40:12.323673 9754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65c16d2756f34f1482b88938a70be0d2" candidate_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7dc901b82a487994426957fd003853"
I20260924 10:40:12.323669 10033 raft_consensus.cc:3037] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:12.323767 9754 raft_consensus.cc:3037] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:12.324762 10033 raft_consensus.cc:2454] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a3be87ca68e428b99d5a6ed98b83c66 in term 1.
I20260924 10:40:12.324853 9754 raft_consensus.cc:2454] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a3be87ca68e428b99d5a6ed98b83c66 in term 1.
I20260924 10:40:12.325162 9829 leader_election.cc:302] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a3be87ca68e428b99d5a6ed98b83c66, 6b7dc901b82a487994426957fd003853; no voters:
I20260924 10:40:12.325347 10116 raft_consensus.cc:2781] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:12.325547 10116 raft_consensus.cc:686] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 LEADER]: Becoming Leader. State: Replica: 0a3be87ca68e428b99d5a6ed98b83c66, State: Running, Role: LEADER
I20260924 10:40:12.325862 10116 consensus_queue.cc:237] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } }
I20260924 10:40:12.328871 9599 catalog_manager.cc:5495] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a3be87ca68e428b99d5a6ed98b83c66 (127.2.8.2). New cstate: current_term: 1 leader_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } health_report { overall_health: HEALTHY } } }
I20260924 10:40:12.394091 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.390664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.2.8.2:40615)
0924 10:40:12.390792 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:12.390796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:12.390811 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:40:12.390966 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:12.390969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:12.391024 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:12.391109 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:40:12.391316 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.391848 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.392705 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.392906 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.393052 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.393273 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:40:12.393694 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:40:12.393776 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:12.393857 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 10:40:12.393902 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:40:12.400008 10033 raft_consensus.cc:1261] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406 [term 1 FOLLOWER]: Refusing update from remote peer 0a3be87ca68e428b99d5a6ed98b83c66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:12.400046 9754 raft_consensus.cc:1261] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Refusing update from remote peer 0a3be87ca68e428b99d5a6ed98b83c66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:12.400728 10121 consensus_queue.cc:1035] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:12.400923 10116 consensus_queue.cc:1035] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:12.407918 10124 mvcc.cc:205] Tried to move back new op lower bound from 7332849305183125504 to 7332849304889098240. Current Snapshot: MvccSnapshot[applied={T|T < 7332849305183125504}]
I20260924 10:40:12.410647 10125 mvcc.cc:205] Tried to move back new op lower bound from 7332849305183125504 to 7332849304889098240. Current Snapshot: MvccSnapshot[applied={T|T < 7332849305183125504}]
I20260924 10:40:12.410773 10126 mvcc.cc:205] Tried to move back new op lower bound from 7332849305183125504 to 7332849304889098240. Current Snapshot: MvccSnapshot[applied={T|T < 7332849305183125504 or (T in {7332849305183125504})}]
I20260924 10:40:12.416020 9599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:40:12.416158 9599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:40:12.421391 9599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66: Sending DeleteTablet for 3 replicas of tablet 65c16d2756f34f1482b88938a70be0d2
I20260924 10:40:12.422169 9875 tablet_service.cc:1514] Processing DeleteTablet for tablet 65c16d2756f34f1482b88938a70be0d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:12 UTC) from {username='oryx', principal='oryx/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:44094
I20260924 10:40:12.422415 10006 tablet_service.cc:1514] Processing DeleteTablet for tablet 65c16d2756f34f1482b88938a70be0d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:12 UTC) from {username='oryx', principal='oryx/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:36218
I20260924 10:40:12.422698 10134 tablet_replica.cc:330] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66: stopping tablet replica
I20260924 10:40:12.422824 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet 65c16d2756f34f1482b88938a70be0d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:12 UTC) from {username='oryx', principal='oryx/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:54274
I20260924 10:40:12.422994 10134 raft_consensus.cc:2229] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:40:12.423312 10134 raft_consensus.cc:2258] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:12.424005 10136 tablet_replica.cc:330] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853: stopping tablet replica
I20260924 10:40:12.424243 10136 raft_consensus.cc:2229] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:40:12.424220 9599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 10:40:12.424501 10136 raft_consensus.cc:2258] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 10:40:12.424618 9599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:40:12.424748 10135 tablet_replica.cc:330] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406: stopping tablet replica
I20260924 10:40:12.425009 10135 raft_consensus.cc:2229] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:40:12.425226 10135 raft_consensus.cc:2258] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:12.426528 10135 ts_tablet_manager.cc:1905] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:12.428821 10134 ts_tablet_manager.cc:1905] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:12.429279 10135 ts_tablet_manager.cc:1918] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:40:12.429378 10135 log.cc:1196] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/65c16d2756f34f1482b88938a70be0d2
I20260924 10:40:12.429687 10135 ts_tablet_manager.cc:1939] T 65c16d2756f34f1482b88938a70be0d2 P 0dadee832c554cd9bb35d52d6d608406: Deleting consensus metadata
I20260924 10:40:12.430550 10136 ts_tablet_manager.cc:1905] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:12.430892 10006 tablet_service.cc:1467] Processing CreateTablet for tablet ef1c57b485944b04a61abc9d2899dbe8 (DEFAULT_TABLE table=test-table [id=b4280c308f284b86a4d9a9c498128619]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:12.431128 10006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1c57b485944b04a61abc9d2899dbe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:12.431092 9875 tablet_service.cc:1467] Processing CreateTablet for tablet ef1c57b485944b04a61abc9d2899dbe8 (DEFAULT_TABLE table=test-table [id=b4280c308f284b86a4d9a9c498128619]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:12.431301 9875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1c57b485944b04a61abc9d2899dbe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:12.432224 9585 catalog_manager.cc:4841] TS 0dadee832c554cd9bb35d52d6d608406 (127.2.8.3:39475): tablet 65c16d2756f34f1482b88938a70be0d2 (table test-table [id=1b4d0034df35423eac79e04a6afa6d0d]) successfully deleted
I20260924 10:40:12.430605 9736 tablet_service.cc:1467] Processing CreateTablet for tablet ef1c57b485944b04a61abc9d2899dbe8 (DEFAULT_TABLE table=test-table [id=b4280c308f284b86a4d9a9c498128619]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:12.432761 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1c57b485944b04a61abc9d2899dbe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:12.433876 10110 tablet_bootstrap.cc:492] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66: Bootstrap starting.
I20260924 10:40:12.434897 10110 tablet_bootstrap.cc:654] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:12.435848 10136 ts_tablet_manager.cc:1918] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:40:12.436223 10134 ts_tablet_manager.cc:1918] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:40:12.436389 10134 log.cc:1196] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/65c16d2756f34f1482b88938a70be0d2
I20260924 10:40:12.436757 10134 ts_tablet_manager.cc:1939] T 65c16d2756f34f1482b88938a70be0d2 P 0a3be87ca68e428b99d5a6ed98b83c66: Deleting consensus metadata
I20260924 10:40:12.437752 10111 tablet_bootstrap.cc:492] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406: Bootstrap starting.
I20260924 10:40:12.437963 9586 catalog_manager.cc:4841] TS 0a3be87ca68e428b99d5a6ed98b83c66 (127.2.8.2:40615): tablet 65c16d2756f34f1482b88938a70be0d2 (table test-table [id=1b4d0034df35423eac79e04a6afa6d0d]) successfully deleted
I20260924 10:40:12.438171 10136 log.cc:1196] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/wal/wals/65c16d2756f34f1482b88938a70be0d2
I20260924 10:40:12.438521 10136 ts_tablet_manager.cc:1939] T 65c16d2756f34f1482b88938a70be0d2 P 6b7dc901b82a487994426957fd003853: Deleting consensus metadata
I20260924 10:40:12.438622 10111 tablet_bootstrap.cc:654] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:12.439898 9585 catalog_manager.cc:4841] TS 6b7dc901b82a487994426957fd003853 (127.2.8.1:45945): tablet 65c16d2756f34f1482b88938a70be0d2 (table test-table [id=1b4d0034df35423eac79e04a6afa6d0d]) successfully deleted
I20260924 10:40:12.440198 10112 tablet_bootstrap.cc:492] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853: Bootstrap starting.
I20260924 10:40:12.440774 10110 tablet_bootstrap.cc:492] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66: No bootstrap required, opened a new log
I20260924 10:40:12.440858 10110 ts_tablet_manager.cc:1397] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260924 10:40:12.440881 10111 tablet_bootstrap.cc:492] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406: No bootstrap required, opened a new log
I20260924 10:40:12.440966 10111 ts_tablet_manager.cc:1397] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 10:40:12.441071 10112 tablet_bootstrap.cc:654] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:12.441336 10110 raft_consensus.cc:348] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.441447 10110 raft_consensus.cc:374] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:12.441479 10110 raft_consensus.cc:729] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a3be87ca68e428b99d5a6ed98b83c66, State: Initialized, Role: FOLLOWER
I20260924 10:40:12.441455 10111 raft_consensus.cc:348] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.441570 10111 raft_consensus.cc:374] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:12.441604 10111 raft_consensus.cc:729] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dadee832c554cd9bb35d52d6d608406, State: Initialized, Role: FOLLOWER
I20260924 10:40:12.441599 10110 consensus_queue.cc:260] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.441735 10111 consensus_queue.cc:260] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.441823 10110 ts_tablet_manager.cc:1428] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:40:12.442219 10112 tablet_bootstrap.cc:492] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853: No bootstrap required, opened a new log
I20260924 10:40:12.442301 10112 ts_tablet_manager.cc:1397] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 10:40:12.442705 10112 raft_consensus.cc:348] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.442788 10111 ts_tablet_manager.cc:1428] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 10:40:12.442826 10112 raft_consensus.cc:374] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:12.442858 10112 raft_consensus.cc:729] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7dc901b82a487994426957fd003853, State: Initialized, Role: FOLLOWER
I20260924 10:40:12.442974 10112 consensus_queue.cc:260] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.443202 10112 ts_tablet_manager.cc:1428] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:40:12.449426 10117 raft_consensus.cc:482] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:40:12.449544 10117 raft_consensus.cc:504] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.450562 10117 leader_election.cc:288] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7dc901b82a487994426957fd003853 (127.2.8.1:45945), 0a3be87ca68e428b99d5a6ed98b83c66 (127.2.8.2:40615)
I20260924 10:40:12.455546 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.450842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:55577 (local address 127.2.8.2:40615)
0924 10:40:12.450971 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:12.450975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:12.451279 (+ 304us) server_negotiation.cc:407] Connection header received
0924 10:40:12.451463 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:12.451466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:12.451510 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:12.451585 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:12.451932 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.452613 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.454414 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.455030 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.455137 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.455199 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:12.455366 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 10:40:12.455435 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:40:12.455597 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.450785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:40615 (local address 127.2.8.3:55577)
0924 10:40:12.451197 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:12.451212 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:12.451307 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:12.451602 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:12.451604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:12.451615 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:12.451745 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.451753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.452744 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.452750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.454291 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.454301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.455152 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.455155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.455232 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.455253 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:40:12.455348 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 10:40:12.455418 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":124,"thread_start_us":94,"threads_started":1}
I20260924 10:40:12.456120 9878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1c57b485944b04a61abc9d2899dbe8" candidate_uuid: "0dadee832c554cd9bb35d52d6d608406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" is_pre_election: true
I20260924 10:40:12.456357 9878 raft_consensus.cc:2454] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dadee832c554cd9bb35d52d6d608406 in term 0.
I20260924 10:40:12.456532 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.450696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:45945 (local address 127.2.8.3:40201)
0924 10:40:12.451025 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:12.451039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:12.451132 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:12.451504 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:12.451507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:12.451520 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:12.451701 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.451711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.452798 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.452801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.454291 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.454301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.456093 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.456097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.456208 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.456229 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:40:12.456329 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 10:40:12.456394 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":159,"thread_start_us":55,"threads_started":1}
I20260924 10:40:12.456606 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.451058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:40201 (local address 127.2.8.1:45945)
0924 10:40:12.451169 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:12.451173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:12.451187 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:40:12.451239 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:12.451245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:12.451292 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:12.451364 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:12.451877 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.452712 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.455255 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.455870 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.455980 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.456036 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:12.456376 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 10:40:12.456489 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:40:12.456826 9970 leader_election.cc:302] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a3be87ca68e428b99d5a6ed98b83c66, 0dadee832c554cd9bb35d52d6d608406; no voters:
I20260924 10:40:12.457290 10117 raft_consensus.cc:2781] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:40:12.457274 9754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1c57b485944b04a61abc9d2899dbe8" candidate_uuid: "0dadee832c554cd9bb35d52d6d608406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7dc901b82a487994426957fd003853" is_pre_election: true
I20260924 10:40:12.457353 10117 raft_consensus.cc:482] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:40:12.457397 10117 raft_consensus.cc:3037] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:12.457392 9754 raft_consensus.cc:2454] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dadee832c554cd9bb35d52d6d608406 in term 0.
I20260924 10:40:12.458422 10117 raft_consensus.cc:504] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.458786 10117 leader_election.cc:288] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [CANDIDATE]: Term 1 election: Requested vote from peers 6b7dc901b82a487994426957fd003853 (127.2.8.1:45945), 0a3be87ca68e428b99d5a6ed98b83c66 (127.2.8.2:40615)
I20260924 10:40:12.459067 9754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1c57b485944b04a61abc9d2899dbe8" candidate_uuid: "0dadee832c554cd9bb35d52d6d608406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7dc901b82a487994426957fd003853"
I20260924 10:40:12.459164 9754 raft_consensus.cc:3037] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:12.459354 9878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1c57b485944b04a61abc9d2899dbe8" candidate_uuid: "0dadee832c554cd9bb35d52d6d608406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a3be87ca68e428b99d5a6ed98b83c66"
I20260924 10:40:12.459468 9878 raft_consensus.cc:3037] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:12.460227 9754 raft_consensus.cc:2454] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dadee832c554cd9bb35d52d6d608406 in term 1.
I20260924 10:40:12.460366 9878 raft_consensus.cc:2454] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dadee832c554cd9bb35d52d6d608406 in term 1.
I20260924 10:40:12.460554 9969 leader_election.cc:302] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dadee832c554cd9bb35d52d6d608406, 6b7dc901b82a487994426957fd003853; no voters:
I20260924 10:40:12.460711 10117 raft_consensus.cc:2781] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:12.460927 10117 raft_consensus.cc:686] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [term 1 LEADER]: Becoming Leader. State: Replica: 0dadee832c554cd9bb35d52d6d608406, State: Running, Role: LEADER
I20260924 10:40:12.461203 10117 consensus_queue.cc:237] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.463227 9597 catalog_manager.cc:5495] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dadee832c554cd9bb35d52d6d608406 (127.2.8.3). New cstate: current_term: 1 leader_uuid: "0dadee832c554cd9bb35d52d6d608406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } health_report { overall_health: HEALTHY } } }
Sep 24 10:40:12 dist-test-slave-j38h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246411, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.62@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM
I20260924 10:40:12.495481 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.486700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:39019 (local address 127.0.0.1:40264)
0924 10:40:12.487012 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:12.487028 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:12.487188 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:12.487460 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:12.487463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:12.487892 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:12.487978 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.487985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.488748 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.488752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.489382 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.489391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.489801 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.489821 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.489888 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.490495 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:12.490507 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:12.492084 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:12.493935 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:12.493943 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:12.493949 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:12.494167 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:12.494442 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:12.494445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:12.494447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:12.494525 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:12.494889 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:12.494897 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:12.495008 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 10:40:12.495117 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 10:40:12.495266 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":118,"threads_started":1}
I20260924 10:40:12.495481 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.486766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.2.8.62:39019)
0924 10:40:12.487029 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:12.487040 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:12.487125 (+ 85us) server_negotiation.cc:407] Connection header received
0924 10:40:12.487284 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:12.487287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:12.487332 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:12.487446 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:12.488101 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.488636 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.489504 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.489713 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.489816 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.492220 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:12.492226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:12.492233 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:12.492252 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:12.493787 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:12.494264 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:12.494270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:12.494272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:12.494349 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:12.494609 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:12.494614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:12.494616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:12.494787 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:12.494897 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:12.495169 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 10:40:12.495319 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":209,"thread_start_us":131,"threads_started":1}
I20260924 10:40:12.500274 9597 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:40264:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 10:40:12.501106 9597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:40:12.506013 9736 tablet_service.cc:1467] Processing CreateTablet for tablet f7d882a15dec4d4683f7c20c4673494c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8119c23abc8a48dcb09d50d1d602ce00]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:40:12.506220 10006 tablet_service.cc:1467] Processing CreateTablet for tablet f7d882a15dec4d4683f7c20c4673494c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8119c23abc8a48dcb09d50d1d602ce00]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:40:12.506220 9875 tablet_service.cc:1467] Processing CreateTablet for tablet f7d882a15dec4d4683f7c20c4673494c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8119c23abc8a48dcb09d50d1d602ce00]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 10:40:12.506402 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d882a15dec4d4683f7c20c4673494c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:12.506575 10006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d882a15dec4d4683f7c20c4673494c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:12.506575 9875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d882a15dec4d4683f7c20c4673494c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:12.509236 10110 tablet_bootstrap.cc:492] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66: Bootstrap starting.
I20260924 10:40:12.509281 10111 tablet_bootstrap.cc:492] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406: Bootstrap starting.
I20260924 10:40:12.510116 10111 tablet_bootstrap.cc:654] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:12.510113 10110 tablet_bootstrap.cc:654] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:12.510831 10112 tablet_bootstrap.cc:492] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853: Bootstrap starting.
I20260924 10:40:12.511583 10111 tablet_bootstrap.cc:492] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406: No bootstrap required, opened a new log
I20260924 10:40:12.511698 10111 ts_tablet_manager.cc:1397] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 10:40:12.511749 10112 tablet_bootstrap.cc:654] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:12.512087 10111 raft_consensus.cc:348] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.512207 10111 raft_consensus.cc:374] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:12.512246 10111 raft_consensus.cc:729] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dadee832c554cd9bb35d52d6d608406, State: Initialized, Role: FOLLOWER
I20260924 10:40:12.512380 10111 consensus_queue.cc:260] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.512662 10111 ts_tablet_manager.cc:1428] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 10:40:12.512660 10110 tablet_bootstrap.cc:492] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66: No bootstrap required, opened a new log
I20260924 10:40:12.512754 10110 ts_tablet_manager.cc:1397] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 10:40:12.512858 10117 tablet_replica.cc:439] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } }
I20260924 10:40:12.513217 10110 raft_consensus.cc:348] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.513327 10110 raft_consensus.cc:374] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:12.513357 10110 raft_consensus.cc:729] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a3be87ca68e428b99d5a6ed98b83c66, State: Initialized, Role: FOLLOWER
I20260924 10:40:12.513475 10110 consensus_queue.cc:260] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.513674 10117 tablet_replica.cc:442] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:40:12.513746 10110 ts_tablet_manager.cc:1428] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:40:12.513939 10130 tablet_replica.cc:439] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } }
I20260924 10:40:12.514052 10130 tablet_replica.cc:442] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:40:12.515548 10112 tablet_bootstrap.cc:492] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853: No bootstrap required, opened a new log
I20260924 10:40:12.515637 10112 ts_tablet_manager.cc:1397] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260924 10:40:12.516049 10112 raft_consensus.cc:348] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.516161 10112 raft_consensus.cc:374] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:12.516199 10112 raft_consensus.cc:729] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7dc901b82a487994426957fd003853, State: Initialized, Role: FOLLOWER
I20260924 10:40:12.516311 10112 consensus_queue.cc:260] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.516575 10118 tablet_replica.cc:439] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } }
I20260924 10:40:12.516722 10118 tablet_replica.cc:442] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:40:12.516575 10112 ts_tablet_manager.cc:1428] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:40:12.563706 10118 raft_consensus.cc:482] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:40:12.563882 10118 raft_consensus.cc:504] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.565092 10118 leader_election.cc:288] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a3be87ca68e428b99d5a6ed98b83c66 (127.2.8.2:40615), 0dadee832c554cd9bb35d52d6d608406 (127.2.8.3:39475)
I20260924 10:40:12.569437 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.565509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:37325 (local address 127.2.8.3:39475)
0924 10:40:12.565620 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:12.565624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:12.565815 (+ 191us) server_negotiation.cc:407] Connection header received
0924 10:40:12.566017 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:12.566020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:12.566071 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:12.566152 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:12.566507 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.567291 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.568397 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.568843 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.568949 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.569058 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:12.569230 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 10:40:12.569306 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:40:12.569504 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.565330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:39475 (local address 127.2.8.1:37325)
0924 10:40:12.565721 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:12.565737 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:12.565855 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:12.566180 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:12.566185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:12.566210 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:12.566383 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.566390 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.567438 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.567441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.568263 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.568272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.568986 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.568990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.569100 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.569126 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 10:40:12.569201 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 10:40:12.569303 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":29,"thread_start_us":121,"threads_started":1}
I20260924 10:40:12.569998 10033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d882a15dec4d4683f7c20c4673494c" candidate_uuid: "6b7dc901b82a487994426957fd003853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dadee832c554cd9bb35d52d6d608406" is_pre_election: true
I20260924 10:40:12.570140 10033 raft_consensus.cc:2454] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b7dc901b82a487994426957fd003853 in term 0.
I20260924 10:40:12.570173 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.565420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:57545 (local address 127.2.8.2:40615)
0924 10:40:12.565565 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:12.565571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:12.565946 (+ 375us) server_negotiation.cc:407] Connection header received
0924 10:40:12.566076 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:12.566079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:12.566159 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:12.566228 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:12.566556 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.567291 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.568815 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.569428 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.569538 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.569607 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:12.569976 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 10:40:12.570044 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:40:12.570598 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.565333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:40615 (local address 127.2.8.1:57545)
0924 10:40:12.565845 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:12.565856 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:12.565941 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:12.566276 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:12.566279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:12.566297 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:12.566460 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.566466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.567414 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.567417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.568686 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.568697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.569731 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.569737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.569842 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.569864 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:40:12.570370 (+ 506us) client_negotiation.cc:240] Negotiation successful
0924 10:40:12.570451 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":306,"thread_start_us":86,"threads_started":1}
I20260924 10:40:12.570603 9689 leader_election.cc:302] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dadee832c554cd9bb35d52d6d608406, 6b7dc901b82a487994426957fd003853; no voters:
I20260924 10:40:12.570921 10118 raft_consensus.cc:2781] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:40:12.571017 10118 raft_consensus.cc:482] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:40:12.571066 10118 raft_consensus.cc:3037] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:12.571142 9878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d882a15dec4d4683f7c20c4673494c" candidate_uuid: "6b7dc901b82a487994426957fd003853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" is_pre_election: true
I20260924 10:40:12.571272 9878 raft_consensus.cc:2454] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b7dc901b82a487994426957fd003853 in term 0.
I20260924 10:40:12.572110 10118 raft_consensus.cc:504] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.572476 10118 leader_election.cc:288] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [CANDIDATE]: Term 1 election: Requested vote from peers 0a3be87ca68e428b99d5a6ed98b83c66 (127.2.8.2:40615), 0dadee832c554cd9bb35d52d6d608406 (127.2.8.3:39475)
I20260924 10:40:12.572784 9878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d882a15dec4d4683f7c20c4673494c" candidate_uuid: "6b7dc901b82a487994426957fd003853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a3be87ca68e428b99d5a6ed98b83c66"
I20260924 10:40:12.572820 10033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d882a15dec4d4683f7c20c4673494c" candidate_uuid: "6b7dc901b82a487994426957fd003853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dadee832c554cd9bb35d52d6d608406"
I20260924 10:40:12.572942 10033 raft_consensus.cc:3037] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:12.572937 9878 raft_consensus.cc:3037] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:12.573921 10033 raft_consensus.cc:2454] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7dc901b82a487994426957fd003853 in term 1.
I20260924 10:40:12.574139 9878 raft_consensus.cc:2454] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7dc901b82a487994426957fd003853 in term 1.
I20260924 10:40:12.574183 9689 leader_election.cc:302] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dadee832c554cd9bb35d52d6d608406, 6b7dc901b82a487994426957fd003853; no voters:
I20260924 10:40:12.574340 10118 raft_consensus.cc:2781] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:12.574581 10118 raft_consensus.cc:686] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [term 1 LEADER]: Becoming Leader. State: Replica: 6b7dc901b82a487994426957fd003853, State: Running, Role: LEADER
I20260924 10:40:12.574851 10118 consensus_queue.cc:237] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.576709 10156 tablet_replica.cc:439] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853: TxnStatusTablet state changed. Reason: New leader 6b7dc901b82a487994426957fd003853. Latest consensus state: current_term: 1 leader_uuid: "6b7dc901b82a487994426957fd003853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } }
I20260924 10:40:12.576786 10156 tablet_replica.cc:442] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:40:12.577034 10158 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 10:40:12.577154 10158 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 10:40:12.577160 9597 catalog_manager.cc:5495] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b7dc901b82a487994426957fd003853 (127.2.8.1). New cstate: current_term: 1 leader_uuid: "6b7dc901b82a487994426957fd003853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } health_report { overall_health: UNKNOWN } } }
I20260924 10:40:12.601413 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.597148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:45945 (local address 127.0.0.1:54280)
0924 10:40:12.597281 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:12.597296 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:12.597390 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:12.597737 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:12.597740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:12.597750 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:40:12.597885 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.597891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.598985 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.598988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.599710 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.599720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.600267 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.600278 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.600343 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.600383 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:40:12.601136 (+ 753us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:40:12.601147 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 10:40:12.601196 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 10:40:12.601288 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 10:40:12.601433 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.597238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54280 (local address 127.2.8.1:45945)
0924 10:40:12.597363 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:12.597368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:12.597385 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:40:12.597541 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:12.597544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:12.597608 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:12.597699 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:40:12.598048 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.598857 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.599816 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.600169 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.600272 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.600521 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:40:12.601043 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:40:12.601144 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:12.601211 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 10:40:12.601283 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 10:40:12.604326 9878 raft_consensus.cc:1261] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Refusing update from remote peer 6b7dc901b82a487994426957fd003853: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:12.604830 10033 raft_consensus.cc:1261] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406 [term 1 FOLLOWER]: Refusing update from remote peer 6b7dc901b82a487994426957fd003853: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:12.604918 10156 consensus_queue.cc:1035] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:12.605315 10118 consensus_queue.cc:1035] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:12.606590 10117 tablet_replica.cc:439] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406: TxnStatusTablet state changed. Reason: New leader 6b7dc901b82a487994426957fd003853. Latest consensus state: current_term: 1 leader_uuid: "6b7dc901b82a487994426957fd003853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } }
I20260924 10:40:12.606701 10117 tablet_replica.cc:442] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:40:12.606925 10130 tablet_replica.cc:439] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66: TxnStatusTablet state changed. Reason: New leader 6b7dc901b82a487994426957fd003853. Latest consensus state: current_term: 1 leader_uuid: "6b7dc901b82a487994426957fd003853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } }
I20260924 10:40:12.607034 10130 tablet_replica.cc:442] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:40:12.609145 10156 tablet_replica.cc:439] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b7dc901b82a487994426957fd003853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } }
I20260924 10:40:12.609252 10156 tablet_replica.cc:442] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:40:12.610711 10130 tablet_replica.cc:439] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b7dc901b82a487994426957fd003853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } }
I20260924 10:40:12.610812 10130 tablet_replica.cc:442] T f7d882a15dec4d4683f7c20c4673494c P 0a3be87ca68e428b99d5a6ed98b83c66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:40:12.614368 10156 tablet_replica.cc:439] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b7dc901b82a487994426957fd003853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } }
I20260924 10:40:12.614485 10156 tablet_replica.cc:442] T f7d882a15dec4d4683f7c20c4673494c P 6b7dc901b82a487994426957fd003853: This TxnStatusTablet replica's current role is: LEADER
I20260924 10:40:12.615440 10117 tablet_replica.cc:439] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b7dc901b82a487994426957fd003853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } } }
I20260924 10:40:12.615558 10117 tablet_replica.cc:442] T f7d882a15dec4d4683f7c20c4673494c P 0dadee832c554cd9bb35d52d6d608406: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 10:40:12.617585 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.613128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.2.8.3:39475)
0924 10:40:12.613231 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:12.613236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:12.613330 (+ 94us) server_negotiation.cc:407] Connection header received
0924 10:40:12.613520 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:12.613523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:12.613584 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:12.613690 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:40:12.613946 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.614712 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.615814 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.616132 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.616223 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.616600 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:40:12.617174 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:40:12.617282 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:12.617388 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 10:40:12.617435 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 10:40:12.628666 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.624634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:52967 (local address 127.2.8.1:45945)
0924 10:40:12.624766 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:12.624771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:12.624901 (+ 130us) server_negotiation.cc:407] Connection header received
0924 10:40:12.625054 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:12.625057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:12.625100 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:12.625187 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:40:12.625455 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.625937 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.627349 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.627688 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.627786 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.628159 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:40:12.628238 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:40:12.628304 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:12.628478 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 10:40:12.628528 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:40:12.628805 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.624553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:45945 (local address 127.2.8.3:52967)
0924 10:40:12.624807 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:12.624821 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:12.624948 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:12.625195 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:12.625198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:12.625205 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:40:12.625345 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.625351 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.626037 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.626044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.627224 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.627232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.627854 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.627870 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.627957 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.628003 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:40:12.628348 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:40:12.628355 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:40:12.628449 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 10:40:12.628557 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":84,"threads_started":1}
I20260924 10:40:12.637429 9878 raft_consensus.cc:1261] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Refusing update from remote peer 0dadee832c554cd9bb35d52d6d608406: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:12.637576 9754 raft_consensus.cc:1261] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Refusing update from remote peer 0dadee832c554cd9bb35d52d6d608406: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:12.637964 10117 consensus_queue.cc:1035] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:12.638300 10117 consensus_queue.cc:1035] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:12.663910 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.660176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:57661 (local address 127.2.8.3:39475)
0924 10:40:12.660335 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:12.660338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:12.660457 (+ 119us) server_negotiation.cc:407] Connection header received
0924 10:40:12.660606 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:12.660609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:12.660646 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:12.660706 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:40:12.661022 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.661739 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.662816 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.663068 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.663179 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.663473 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:40:12.663553 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:40:12.663631 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:12.663723 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 10:40:12.663771 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 10:40:12.664093 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.660055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:39475 (local address 127.2.8.1:57661)
0924 10:40:12.660386 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:12.660401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:12.660527 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:12.660738 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:12.660741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:12.660748 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 10:40:12.660869 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.660874 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.661856 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.661859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.662715 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:12.662726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.663178 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.663193 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:12.663293 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.663337 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 10:40:12.663646 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 10:40:12.663654 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:40:12.663714 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 10:40:12.663789 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":88,"threads_started":1}
I20260924 10:40:12.691408 9597 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:40:12.691606 9597 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:40:12.693953 9597 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66: Sending DeleteTablet for 3 replicas of tablet ef1c57b485944b04a61abc9d2899dbe8
I20260924 10:40:12.694669 9875 tablet_service.cc:1514] Processing DeleteTablet for tablet ef1c57b485944b04a61abc9d2899dbe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:12 UTC) from {username='oryx', principal='oryx/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:44094
I20260924 10:40:12.694865 10006 tablet_service.cc:1514] Processing DeleteTablet for tablet ef1c57b485944b04a61abc9d2899dbe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:12 UTC) from {username='oryx', principal='oryx/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:36218
I20260924 10:40:12.695030 10135 tablet_replica.cc:330] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406: stopping tablet replica
I20260924 10:40:12.695048 10134 tablet_replica.cc:330] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66: stopping tablet replica
I20260924 10:40:12.695154 10135 raft_consensus.cc:2229] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:40:12.695153 10134 raft_consensus.cc:2229] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:40:12.695272 10134 raft_consensus.cc:2258] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:12.695457 10135 raft_consensus.cc:2258] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:12.696151 10134 ts_tablet_manager.cc:1905] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:12.696213 10135 ts_tablet_manager.cc:1905] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:12.698577 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet ef1c57b485944b04a61abc9d2899dbe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:12 UTC) from {username='oryx', principal='oryx/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:54274
I20260924 10:40:12.698750 10136 tablet_replica.cc:330] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853: stopping tablet replica
I20260924 10:40:12.698849 10136 raft_consensus.cc:2229] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:40:12.698956 10136 raft_consensus.cc:2258] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:12.699195 10134 ts_tablet_manager.cc:1918] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:40:12.699268 10134 log.cc:1196] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/ef1c57b485944b04a61abc9d2899dbe8
I20260924 10:40:12.699538 10134 ts_tablet_manager.cc:1939] T ef1c57b485944b04a61abc9d2899dbe8 P 0a3be87ca68e428b99d5a6ed98b83c66: Deleting consensus metadata
I20260924 10:40:12.699815 10136 ts_tablet_manager.cc:1905] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:12.700263 9586 catalog_manager.cc:4841] TS 0a3be87ca68e428b99d5a6ed98b83c66 (127.2.8.2:40615): tablet ef1c57b485944b04a61abc9d2899dbe8 (table test-table [id=b4280c308f284b86a4d9a9c498128619]) successfully deleted
I20260924 10:40:12.703063 10136 ts_tablet_manager.cc:1918] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:40:12.703153 10136 log.cc:1196] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/wal/wals/ef1c57b485944b04a61abc9d2899dbe8
I20260924 10:40:12.703433 10136 ts_tablet_manager.cc:1939] T ef1c57b485944b04a61abc9d2899dbe8 P 6b7dc901b82a487994426957fd003853: Deleting consensus metadata
I20260924 10:40:12.704156 9585 catalog_manager.cc:4841] TS 6b7dc901b82a487994426957fd003853 (127.2.8.1:45945): tablet ef1c57b485944b04a61abc9d2899dbe8 (table test-table [id=b4280c308f284b86a4d9a9c498128619]) successfully deleted
I20260924 10:40:12.704517 10135 ts_tablet_manager.cc:1918] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:40:12.704625 10135 log.cc:1196] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/ef1c57b485944b04a61abc9d2899dbe8
I20260924 10:40:12.704859 10135 ts_tablet_manager.cc:1939] T ef1c57b485944b04a61abc9d2899dbe8 P 0dadee832c554cd9bb35d52d6d608406: Deleting consensus metadata
I20260924 10:40:12.705540 9585 catalog_manager.cc:4841] TS 0dadee832c554cd9bb35d52d6d608406 (127.2.8.3:39475): tablet ef1c57b485944b04a61abc9d2899dbe8 (table test-table [id=b4280c308f284b86a4d9a9c498128619]) successfully deleted
Sep 24 10:40:12 dist-test-slave-j38h krb5kdc[9554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 10:40:12 dist-test-slave-j38h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:12 dist-test-slave-j38h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:40:12.729692 10150 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:12.721691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.2.8.62:39019)
0924 10:40:12.721817 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:12.721822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:12.722094 (+ 272us) server_negotiation.cc:407] Connection header received
0924 10:40:12.722236 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:12.722240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:12.722295 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:12.722385 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:12.723309 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.724186 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.725420 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.725772 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.725873 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.729562 (+ 3689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40276: BlockingRecv error: recv got EOF from 127.0.0.1:40276 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 10:40:12 dist-test-slave-j38h krb5kdc[9554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM
I20260924 10:40:12.741798 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.731553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.2.8.62:39019)
0924 10:40:12.731654 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:12.731658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:12.732082 (+ 424us) server_negotiation.cc:407] Connection header received
0924 10:40:12.732199 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:12.732202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:12.732247 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:12.732326 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:12.733086 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.733855 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.734957 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.735288 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.735388 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.738190 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:12.738194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:12.738197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:12.738224 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:12.739967 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:12.740481 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:12.740484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:12.740486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:12.740572 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:12.740873 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:12.740876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:12.740878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:12.741120 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:12.741253 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:12.741510 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 10:40:12.741632 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:40:12.744916 9597 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:40:12.745389 9597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:40:12.750334 9736 tablet_service.cc:1467] Processing CreateTablet for tablet a0b0bab257354463bb9ad61b71ee69ee (DEFAULT_TABLE table=test-table [id=7528b6d42ec24282b44ea5706f4fcaf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:12.750646 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b0bab257354463bb9ad61b71ee69ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:12.750468 9875 tablet_service.cc:1467] Processing CreateTablet for tablet a0b0bab257354463bb9ad61b71ee69ee (DEFAULT_TABLE table=test-table [id=7528b6d42ec24282b44ea5706f4fcaf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:12.751873 9875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b0bab257354463bb9ad61b71ee69ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:12.753072 10112 tablet_bootstrap.cc:492] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853: Bootstrap starting.
I20260924 10:40:12.753896 10112 tablet_bootstrap.cc:654] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:12.754179 10110 tablet_bootstrap.cc:492] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66: Bootstrap starting.
I20260924 10:40:12.754956 10110 tablet_bootstrap.cc:654] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:12.754886 10006 tablet_service.cc:1467] Processing CreateTablet for tablet a0b0bab257354463bb9ad61b71ee69ee (DEFAULT_TABLE table=test-table [id=7528b6d42ec24282b44ea5706f4fcaf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:12.755132 10006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b0bab257354463bb9ad61b71ee69ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:12.757408 10111 tablet_bootstrap.cc:492] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406: Bootstrap starting.
I20260924 10:40:12.757661 10110 tablet_bootstrap.cc:492] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66: No bootstrap required, opened a new log
I20260924 10:40:12.757732 10110 ts_tablet_manager.cc:1397] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 10:40:12.758117 10110 raft_consensus.cc:348] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.758222 10110 raft_consensus.cc:374] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:12.758241 10111 tablet_bootstrap.cc:654] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:12.758251 10110 raft_consensus.cc:729] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a3be87ca68e428b99d5a6ed98b83c66, State: Initialized, Role: FOLLOWER
I20260924 10:40:12.758366 10110 consensus_queue.cc:260] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.758612 10110 ts_tablet_manager.cc:1428] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:40:12.759689 10112 tablet_bootstrap.cc:492] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853: No bootstrap required, opened a new log
I20260924 10:40:12.759770 10112 ts_tablet_manager.cc:1397] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260924 10:40:12.760571 10112 raft_consensus.cc:348] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.760679 10112 raft_consensus.cc:374] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:12.760705 10112 raft_consensus.cc:729] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7dc901b82a487994426957fd003853, State: Initialized, Role: FOLLOWER
I20260924 10:40:12.760818 10112 consensus_queue.cc:260] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.762349 10112 ts_tablet_manager.cc:1428] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260924 10:40:12.763422 10111 tablet_bootstrap.cc:492] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406: No bootstrap required, opened a new log
I20260924 10:40:12.763507 10111 ts_tablet_manager.cc:1397] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260924 10:40:12.763942 10111 raft_consensus.cc:348] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.764052 10111 raft_consensus.cc:374] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:12.764087 10111 raft_consensus.cc:729] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dadee832c554cd9bb35d52d6d608406, State: Initialized, Role: FOLLOWER
I20260924 10:40:12.764206 10111 consensus_queue.cc:260] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.764447 10111 ts_tablet_manager.cc:1428] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260924 10:40:12.802554 10117 raft_consensus.cc:482] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:40:12.802707 10117 raft_consensus.cc:504] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.803200 10117 leader_election.cc:288] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7dc901b82a487994426957fd003853 (127.2.8.1:45945), 0a3be87ca68e428b99d5a6ed98b83c66 (127.2.8.2:40615)
I20260924 10:40:12.803613 9880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b0bab257354463bb9ad61b71ee69ee" candidate_uuid: "0dadee832c554cd9bb35d52d6d608406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" is_pre_election: true
I20260924 10:40:12.803622 9754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b0bab257354463bb9ad61b71ee69ee" candidate_uuid: "0dadee832c554cd9bb35d52d6d608406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7dc901b82a487994426957fd003853" is_pre_election: true
I20260924 10:40:12.803766 9754 raft_consensus.cc:2454] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dadee832c554cd9bb35d52d6d608406 in term 0.
I20260924 10:40:12.803771 9880 raft_consensus.cc:2454] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dadee832c554cd9bb35d52d6d608406 in term 0.
I20260924 10:40:12.804039 9970 leader_election.cc:302] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a3be87ca68e428b99d5a6ed98b83c66, 0dadee832c554cd9bb35d52d6d608406; no voters:
I20260924 10:40:12.804212 10117 raft_consensus.cc:2781] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:40:12.804282 10117 raft_consensus.cc:482] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:40:12.804330 10117 raft_consensus.cc:3037] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:12.805619 10117 raft_consensus.cc:504] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.805981 10117 leader_election.cc:288] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [CANDIDATE]: Term 1 election: Requested vote from peers 6b7dc901b82a487994426957fd003853 (127.2.8.1:45945), 0a3be87ca68e428b99d5a6ed98b83c66 (127.2.8.2:40615)
I20260924 10:40:12.806308 9754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b0bab257354463bb9ad61b71ee69ee" candidate_uuid: "0dadee832c554cd9bb35d52d6d608406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7dc901b82a487994426957fd003853"
I20260924 10:40:12.806305 9880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b0bab257354463bb9ad61b71ee69ee" candidate_uuid: "0dadee832c554cd9bb35d52d6d608406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a3be87ca68e428b99d5a6ed98b83c66"
I20260924 10:40:12.806424 9754 raft_consensus.cc:3037] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:12.806445 9880 raft_consensus.cc:3037] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:12.807406 9880 raft_consensus.cc:2454] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dadee832c554cd9bb35d52d6d608406 in term 1.
I20260924 10:40:12.807616 9754 raft_consensus.cc:2454] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dadee832c554cd9bb35d52d6d608406 in term 1.
I20260924 10:40:12.807710 9970 leader_election.cc:302] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a3be87ca68e428b99d5a6ed98b83c66, 0dadee832c554cd9bb35d52d6d608406; no voters:
I20260924 10:40:12.807888 10117 raft_consensus.cc:2781] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:12.808034 10117 raft_consensus.cc:686] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [term 1 LEADER]: Becoming Leader. State: Replica: 0dadee832c554cd9bb35d52d6d608406, State: Running, Role: LEADER
I20260924 10:40:12.808179 10117 consensus_queue.cc:237] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.809979 9597 catalog_manager.cc:5495] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dadee832c554cd9bb35d52d6d608406 (127.2.8.3). New cstate: current_term: 1 leader_uuid: "0dadee832c554cd9bb35d52d6d608406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } health_report { overall_health: HEALTHY } } }
I20260924 10:40:12.821889 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:12.818616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.2.8.3:39475)
0924 10:40:12.818754 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:12.818759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:12.818772 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:40:12.818876 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:12.818878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:12.818930 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:12.819016 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:40:12.819253 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.819754 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.820782 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:12.821073 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:12.821180 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:12.821455 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:40:12.821532 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:40:12.821605 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:12.821706 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 10:40:12.821753 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:40:12.823922 9880 raft_consensus.cc:1261] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Refusing update from remote peer 0dadee832c554cd9bb35d52d6d608406: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:12.823993 9754 raft_consensus.cc:1261] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Refusing update from remote peer 0dadee832c554cd9bb35d52d6d608406: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:12.824411 10117 consensus_queue.cc:1035] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:12.824572 10142 consensus_queue.cc:1035] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:12.833670 9599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:40:12.833931 9599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:40:12.836088 9599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66: Sending DeleteTablet for 3 replicas of tablet a0b0bab257354463bb9ad61b71ee69ee
I20260924 10:40:12.836588 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet a0b0bab257354463bb9ad61b71ee69ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:12 UTC) from {username='oryx', principal='oryx/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:54274
I20260924 10:40:12.836653 9875 tablet_service.cc:1514] Processing DeleteTablet for tablet a0b0bab257354463bb9ad61b71ee69ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:12 UTC) from {username='oryx', principal='oryx/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:44094
I20260924 10:40:12.836786 10136 tablet_replica.cc:330] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853: stopping tablet replica
I20260924 10:40:12.836844 10134 tablet_replica.cc:330] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66: stopping tablet replica
I20260924 10:40:12.836910 10136 raft_consensus.cc:2229] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:40:12.837023 10134 raft_consensus.cc:2229] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:40:12.837061 10136 raft_consensus.cc:2258] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:12.837178 10134 raft_consensus.cc:2258] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:12.837575 10006 tablet_service.cc:1514] Processing DeleteTablet for tablet a0b0bab257354463bb9ad61b71ee69ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:12 UTC) from {username='oryx', principal='oryx/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:36218
I20260924 10:40:12.837738 10135 tablet_replica.cc:330] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406: stopping tablet replica
I20260924 10:40:12.837848 10135 raft_consensus.cc:2229] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:40:12.837939 10134 ts_tablet_manager.cc:1905] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:12.838027 10135 raft_consensus.cc:2258] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:12.838050 10136 ts_tablet_manager.cc:1905] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:12.840442 10134 ts_tablet_manager.cc:1918] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:40:12.840528 10134 log.cc:1196] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/a0b0bab257354463bb9ad61b71ee69ee
I20260924 10:40:12.840721 9599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 10:40:12.840801 10134 ts_tablet_manager.cc:1939] T a0b0bab257354463bb9ad61b71ee69ee P 0a3be87ca68e428b99d5a6ed98b83c66: Deleting consensus metadata
W20260924 10:40:12.841150 9599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:40:12.841526 10136 ts_tablet_manager.cc:1918] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:40:12.841557 9586 catalog_manager.cc:4841] TS 0a3be87ca68e428b99d5a6ed98b83c66 (127.2.8.2:40615): tablet a0b0bab257354463bb9ad61b71ee69ee (table test-table [id=7528b6d42ec24282b44ea5706f4fcaf4]) successfully deleted
I20260924 10:40:12.841603 10136 log.cc:1196] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/wal/wals/a0b0bab257354463bb9ad61b71ee69ee
I20260924 10:40:12.841871 10136 ts_tablet_manager.cc:1939] T a0b0bab257354463bb9ad61b71ee69ee P 6b7dc901b82a487994426957fd003853: Deleting consensus metadata
I20260924 10:40:12.842497 9585 catalog_manager.cc:4841] TS 6b7dc901b82a487994426957fd003853 (127.2.8.1:45945): tablet a0b0bab257354463bb9ad61b71ee69ee (table test-table [id=7528b6d42ec24282b44ea5706f4fcaf4]) successfully deleted
I20260924 10:40:12.843055 10135 ts_tablet_manager.cc:1905] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:12.846751 10135 ts_tablet_manager.cc:1918] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:40:12.846845 10135 log.cc:1196] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/a0b0bab257354463bb9ad61b71ee69ee
I20260924 10:40:12.847107 10135 ts_tablet_manager.cc:1939] T a0b0bab257354463bb9ad61b71ee69ee P 0dadee832c554cd9bb35d52d6d608406: Deleting consensus metadata
I20260924 10:40:12.847741 9585 catalog_manager.cc:4841] TS 0dadee832c554cd9bb35d52d6d608406 (127.2.8.3:39475): tablet a0b0bab257354463bb9ad61b71ee69ee (table test-table [id=7528b6d42ec24282b44ea5706f4fcaf4]) successfully deleted
I20260924 10:40:12.848321 9736 tablet_service.cc:1467] Processing CreateTablet for tablet e44037b1d397410e8d7dbe243f69ea9a (DEFAULT_TABLE table=test-table [id=08f2a43e02b74bb5a7d9c3c3c6a66dea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:12.848546 9875 tablet_service.cc:1467] Processing CreateTablet for tablet e44037b1d397410e8d7dbe243f69ea9a (DEFAULT_TABLE table=test-table [id=08f2a43e02b74bb5a7d9c3c3c6a66dea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:12.848620 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44037b1d397410e8d7dbe243f69ea9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:12.848781 9875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44037b1d397410e8d7dbe243f69ea9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:12.850546 10112 tablet_bootstrap.cc:492] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853: Bootstrap starting.
I20260924 10:40:12.850729 10006 tablet_service.cc:1467] Processing CreateTablet for tablet e44037b1d397410e8d7dbe243f69ea9a (DEFAULT_TABLE table=test-table [id=08f2a43e02b74bb5a7d9c3c3c6a66dea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:12.850914 10110 tablet_bootstrap.cc:492] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66: Bootstrap starting.
I20260924 10:40:12.850941 10006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44037b1d397410e8d7dbe243f69ea9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:12.851383 10112 tablet_bootstrap.cc:654] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:12.853297 10111 tablet_bootstrap.cc:492] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406: Bootstrap starting.
I20260924 10:40:12.854061 10111 tablet_bootstrap.cc:654] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:12.854970 10112 tablet_bootstrap.cc:492] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853: No bootstrap required, opened a new log
I20260924 10:40:12.855088 10112 ts_tablet_manager.cc:1397] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 10:40:12.855525 10112 raft_consensus.cc:348] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.855650 10112 raft_consensus.cc:374] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:12.855691 10112 raft_consensus.cc:729] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7dc901b82a487994426957fd003853, State: Initialized, Role: FOLLOWER
I20260924 10:40:12.855839 10112 consensus_queue.cc:260] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.856127 10112 ts_tablet_manager.cc:1428] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:40:12.857347 10111 tablet_bootstrap.cc:492] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406: No bootstrap required, opened a new log
I20260924 10:40:12.857455 10111 ts_tablet_manager.cc:1397] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260924 10:40:12.857851 10111 raft_consensus.cc:348] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.857975 10111 raft_consensus.cc:374] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:12.858016 10111 raft_consensus.cc:729] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dadee832c554cd9bb35d52d6d608406, State: Initialized, Role: FOLLOWER
I20260924 10:40:12.858143 10111 consensus_queue.cc:260] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.858424 10111 ts_tablet_manager.cc:1428] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 10:40:12.858920 10110 tablet_bootstrap.cc:654] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:12.860834 10110 tablet_bootstrap.cc:492] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66: No bootstrap required, opened a new log
I20260924 10:40:12.860921 10110 ts_tablet_manager.cc:1397] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260924 10:40:12.861333 10110 raft_consensus.cc:348] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.861445 10110 raft_consensus.cc:374] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:12.861475 10110 raft_consensus.cc:729] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a3be87ca68e428b99d5a6ed98b83c66, State: Initialized, Role: FOLLOWER
I20260924 10:40:12.861590 10110 consensus_queue.cc:260] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.861831 10110 ts_tablet_manager.cc:1428] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:40:12.881451 10194 raft_consensus.cc:482] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:40:12.881574 10194 raft_consensus.cc:504] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.881990 10194 leader_election.cc:288] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7dc901b82a487994426957fd003853 (127.2.8.1:45945), 0a3be87ca68e428b99d5a6ed98b83c66 (127.2.8.2:40615)
I20260924 10:40:12.882320 9754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44037b1d397410e8d7dbe243f69ea9a" candidate_uuid: "0dadee832c554cd9bb35d52d6d608406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7dc901b82a487994426957fd003853" is_pre_election: true
I20260924 10:40:12.882354 9880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44037b1d397410e8d7dbe243f69ea9a" candidate_uuid: "0dadee832c554cd9bb35d52d6d608406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" is_pre_election: true
I20260924 10:40:12.882464 9754 raft_consensus.cc:2454] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dadee832c554cd9bb35d52d6d608406 in term 0.
I20260924 10:40:12.882503 9880 raft_consensus.cc:2454] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dadee832c554cd9bb35d52d6d608406 in term 0.
I20260924 10:40:12.882809 9970 leader_election.cc:302] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a3be87ca68e428b99d5a6ed98b83c66, 0dadee832c554cd9bb35d52d6d608406; no voters:
I20260924 10:40:12.883024 10194 raft_consensus.cc:2781] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:40:12.883078 10194 raft_consensus.cc:482] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:40:12.883112 10194 raft_consensus.cc:3037] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:12.884249 10194 raft_consensus.cc:504] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.884562 10194 leader_election.cc:288] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [CANDIDATE]: Term 1 election: Requested vote from peers 6b7dc901b82a487994426957fd003853 (127.2.8.1:45945), 0a3be87ca68e428b99d5a6ed98b83c66 (127.2.8.2:40615)
I20260924 10:40:12.884847 9754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44037b1d397410e8d7dbe243f69ea9a" candidate_uuid: "0dadee832c554cd9bb35d52d6d608406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7dc901b82a487994426957fd003853"
I20260924 10:40:12.884894 9880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44037b1d397410e8d7dbe243f69ea9a" candidate_uuid: "0dadee832c554cd9bb35d52d6d608406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a3be87ca68e428b99d5a6ed98b83c66"
I20260924 10:40:12.885012 9754 raft_consensus.cc:3037] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:12.885035 9880 raft_consensus.cc:3037] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:12.886097 9880 raft_consensus.cc:2454] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dadee832c554cd9bb35d52d6d608406 in term 1.
I20260924 10:40:12.886097 9754 raft_consensus.cc:2454] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dadee832c554cd9bb35d52d6d608406 in term 1.
I20260924 10:40:12.886389 9969 leader_election.cc:302] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dadee832c554cd9bb35d52d6d608406, 6b7dc901b82a487994426957fd003853; no voters:
I20260924 10:40:12.886574 10194 raft_consensus.cc:2781] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:12.886662 10194 raft_consensus.cc:686] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [term 1 LEADER]: Becoming Leader. State: Replica: 0dadee832c554cd9bb35d52d6d608406, State: Running, Role: LEADER
I20260924 10:40:12.886776 10194 consensus_queue.cc:237] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } }
I20260924 10:40:12.888394 9599 catalog_manager.cc:5495] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dadee832c554cd9bb35d52d6d608406 (127.2.8.3). New cstate: current_term: 1 leader_uuid: "0dadee832c554cd9bb35d52d6d608406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dadee832c554cd9bb35d52d6d608406" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39475 } health_report { overall_health: HEALTHY } } }
I20260924 10:40:12.913173 9880 raft_consensus.cc:1261] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Refusing update from remote peer 0dadee832c554cd9bb35d52d6d608406: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:12.913257 9754 raft_consensus.cc:1261] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Refusing update from remote peer 0dadee832c554cd9bb35d52d6d608406: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:12.913618 10194 consensus_queue.cc:1035] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7dc901b82a487994426957fd003853" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 45945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:12.913784 10117 consensus_queue.cc:1035] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a3be87ca68e428b99d5a6ed98b83c66" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:12.944082 9599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:40:12.944223 9599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:40:12.946130 9599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0afdc6214a9a46cca4cbafacfb870c66: Sending DeleteTablet for 3 replicas of tablet e44037b1d397410e8d7dbe243f69ea9a
I20260924 10:40:12.946651 9736 tablet_service.cc:1514] Processing DeleteTablet for tablet e44037b1d397410e8d7dbe243f69ea9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:12 UTC) from {username='oryx', principal='oryx/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:54274
I20260924 10:40:12.946722 9875 tablet_service.cc:1514] Processing DeleteTablet for tablet e44037b1d397410e8d7dbe243f69ea9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:12 UTC) from {username='oryx', principal='oryx/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:44094
I20260924 10:40:12.946741 10006 tablet_service.cc:1514] Processing DeleteTablet for tablet e44037b1d397410e8d7dbe243f69ea9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:12 UTC) from {username='oryx', principal='oryx/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:36218
I20260924 10:40:12.946882 10134 tablet_replica.cc:330] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66: stopping tablet replica
I20260924 10:40:12.946916 10135 tablet_replica.cc:330] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406: stopping tablet replica
I20260924 10:40:12.946969 10136 tablet_replica.cc:330] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853: stopping tablet replica
I20260924 10:40:12.947023 10135 raft_consensus.cc:2229] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:40:12.947027 10134 raft_consensus.cc:2229] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:40:12.947109 10136 raft_consensus.cc:2229] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:40:12.947137 10134 raft_consensus.cc:2258] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:12.947212 10135 raft_consensus.cc:2258] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:12.947232 10136 raft_consensus.cc:2258] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:12.947961 10136 ts_tablet_manager.cc:1905] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:12.947995 10134 ts_tablet_manager.cc:1905] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:12.947988 10135 ts_tablet_manager.cc:1905] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:12.950083 10136 ts_tablet_manager.cc:1918] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:40:12.950176 10136 log.cc:1196] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-0/wal/wals/e44037b1d397410e8d7dbe243f69ea9a
I20260924 10:40:12.950374 10134 ts_tablet_manager.cc:1918] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:40:12.950374 10135 ts_tablet_manager.cc:1918] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 10:40:12.950455 10134 log.cc:1196] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/e44037b1d397410e8d7dbe243f69ea9a
I20260924 10:40:12.950466 10135 log.cc:1196] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/e44037b1d397410e8d7dbe243f69ea9a
I20260924 10:40:12.950470 10136 ts_tablet_manager.cc:1939] T e44037b1d397410e8d7dbe243f69ea9a P 6b7dc901b82a487994426957fd003853: Deleting consensus metadata
I20260924 10:40:12.950728 10135 ts_tablet_manager.cc:1939] T e44037b1d397410e8d7dbe243f69ea9a P 0dadee832c554cd9bb35d52d6d608406: Deleting consensus metadata
I20260924 10:40:12.950728 10134 ts_tablet_manager.cc:1939] T e44037b1d397410e8d7dbe243f69ea9a P 0a3be87ca68e428b99d5a6ed98b83c66: Deleting consensus metadata
I20260924 10:40:12.951227 9585 catalog_manager.cc:4841] TS 6b7dc901b82a487994426957fd003853 (127.2.8.1:45945): tablet e44037b1d397410e8d7dbe243f69ea9a (table test-table [id=08f2a43e02b74bb5a7d9c3c3c6a66dea]) successfully deleted
I20260924 10:40:12.951433 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 9671
I20260924 10:40:12.953117 9586 catalog_manager.cc:4841] TS 0a3be87ca68e428b99d5a6ed98b83c66 (127.2.8.2:40615): tablet e44037b1d397410e8d7dbe243f69ea9a (table test-table [id=08f2a43e02b74bb5a7d9c3c3c6a66dea]) successfully deleted
I20260924 10:40:12.953349 9585 catalog_manager.cc:4841] TS 0dadee832c554cd9bb35d52d6d608406 (127.2.8.3:39475): tablet e44037b1d397410e8d7dbe243f69ea9a (table test-table [id=08f2a43e02b74bb5a7d9c3c3c6a66dea]) successfully deleted
I20260924 10:40:12.959890 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 9811
I20260924 10:40:12.967087 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 9951
I20260924 10:40:12.974519 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 9567
2026-09-24T10:40:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3315 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:40:13 dist-test-slave-j38h krb5kdc[10203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:40:13 dist-test-slave-j38h krb5kdc[10203](info): set up 2 sockets
Sep 24 10:40:13 dist-test-slave-j38h krb5kdc[10203](info): commencing operation
krb5kdc: starting...
W20260924 10:40:14.021765 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:40:14 dist-test-slave-j38h krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:40:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:40:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:40:14.144739 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:35623
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:41009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:35623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:14.255715 10216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:14.255999 10216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:14.256039 10216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:14.260109 10216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:40:14.260171 10216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:14.260191 10216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:14.260210 10216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:40:14.260224 10216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:40:14.265647 10216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:41009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:35623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:35623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:14.266636 10216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:14.267594 10216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:14.276266 10221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:14.276261 10224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:14.276261 10222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:14.276298 10216 server_base.cc:1034] running on GCE node
I20260924 10:40:14.277102 10216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:14.278288 10216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:14.279484 10216 hybrid_clock.cc:648] HybridClock initialized: now 1790246414279457 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T10:40:14Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1316 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:40:14 dist-test-slave-j38h krb5kdc[10229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:40:14 dist-test-slave-j38h krb5kdc[10229](info): set up 2 sockets
Sep 24 10:40:14 dist-test-slave-j38h krb5kdc[10229](info): commencing operation
krb5kdc: starting...
W20260924 10:40:15.333948 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:40:15 dist-test-slave-j38h krb5kdc[10229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:40:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:40:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:40:15.458611 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:41783
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:41783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:15.568104 10242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:15.568434 10242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:15.568509 10242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:15.572600 10242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:40:15.572711 10242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:15.572749 10242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:15.572788 10242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:40:15.572824 10242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:40:15.578349 10242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:41783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:41783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:15.579478 10242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:15.580544 10242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:15.589640 10248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:15.589640 10247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:15.589740 10250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:15.589933 10242 server_base.cc:1034] running on GCE node
I20260924 10:40:15.590590 10242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:15.591787 10242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:15.592967 10242 hybrid_clock.cc:648] HybridClock initialized: now 1790246415592931 us; error 47 us; skew 500 ppm
Sep 24 10:40:15 dist-test-slave-j38h krb5kdc[10229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246415, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:15.607366 10242 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:40:15.608317 10242 webserver.cc:460] Webserver started at http://127.2.8.62:45677/ using document root <none> and password file <none>
I20260924 10:40:15.608911 10242 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:15.609040 10242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:15.609303 10242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:15.611609 10242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "4439ac941a4e4b7ea031d24cdeb383e6"
format_stamp: "Formatted at 2026-09-24 10:40:15 on dist-test-slave-j38h"
server_key: "40e03079b4ca5975281112f4e04e0e9e"
server_key_iv: "4c389e1a50aedf9b71ee0bd33955cee5"
server_key_version: "encryptionkey@0"
I20260924 10:40:15.612213 10242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "4439ac941a4e4b7ea031d24cdeb383e6"
format_stamp: "Formatted at 2026-09-24 10:40:15 on dist-test-slave-j38h"
server_key: "40e03079b4ca5975281112f4e04e0e9e"
server_key_iv: "4c389e1a50aedf9b71ee0bd33955cee5"
server_key_version: "encryptionkey@0"
I20260924 10:40:15.616113 10242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:40:15.618494 10257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:15.619791 10242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:40:15.619963 10242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "4439ac941a4e4b7ea031d24cdeb383e6"
format_stamp: "Formatted at 2026-09-24 10:40:15 on dist-test-slave-j38h"
server_key: "40e03079b4ca5975281112f4e04e0e9e"
server_key_iv: "4c389e1a50aedf9b71ee0bd33955cee5"
server_key_version: "encryptionkey@0"
I20260924 10:40:15.620105 10242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:15.633072 10242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:15.636111 10242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:15.636315 10242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:15.644229 10242 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:41783
I20260924 10:40:15.644248 10309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:41783 every 8 connection(s)
I20260924 10:40:15.645263 10242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:40:15.647856 10310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:15.653252 10310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6: Bootstrap starting.
I20260924 10:40:15.655287 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 10242
I20260924 10:40:15.655404 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:40:15.655740 2080 external_mini_cluster.cc:1428] Setting key 6aca1a539ee0735f023b38deca6424b4
I20260924 10:40:15.655809 10310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:15.656553 10310 log.cc:824] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:15.658981 10310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6: No bootstrap required, opened a new log
I20260924 10:40:15.663355 10310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ac941a4e4b7ea031d24cdeb383e6" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41783 } }
I20260924 10:40:15.663659 10310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:15.663741 10310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4439ac941a4e4b7ea031d24cdeb383e6, State: Initialized, Role: FOLLOWER
I20260924 10:40:15.664216 10310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ac941a4e4b7ea031d24cdeb383e6" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41783 } }
I20260924 10:40:15.664371 10310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:40:15.664459 10310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:40:15.664580 10310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 10:40:15 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:40:15.665845 10310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ac941a4e4b7ea031d24cdeb383e6" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41783 } }
I20260924 10:40:15.666210 10310 leader_election.cc:302] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4439ac941a4e4b7ea031d24cdeb383e6; no voters:
I20260924 10:40:15.666563 10310 leader_election.cc:288] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:40:15.666688 10315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:15.666987 10315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 1 LEADER]: Becoming Leader. State: Replica: 4439ac941a4e4b7ea031d24cdeb383e6, State: Running, Role: LEADER
I20260924 10:40:15.667457 10315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ac941a4e4b7ea031d24cdeb383e6" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41783 } }
I20260924 10:40:15.667963 10310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:40:15.669332 10317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4439ac941a4e4b7ea031d24cdeb383e6. Latest consensus state: current_term: 1 leader_uuid: "4439ac941a4e4b7ea031d24cdeb383e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ac941a4e4b7ea031d24cdeb383e6" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41783 } } }
I20260924 10:40:15.669461 10317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:15.669823 10316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4439ac941a4e4b7ea031d24cdeb383e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ac941a4e4b7ea031d24cdeb383e6" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41783 } } }
I20260924 10:40:15.669925 10316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:15.672211 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:15.657089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.2.8.62:41783)
0924 10:40:15.657617 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:15.657638 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:15.657680 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:40:15.658541 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:15.658564 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:15.658962 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:15.659330 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:15.660097 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:15.661619 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:15.662545 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:15.663233 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:15.663399 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:15.666431 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:15.666436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:15.666451 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:15.666478 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:15.668678 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:15.669339 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:15.669348 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:15.669356 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:15.669499 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:15.669887 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:15.669892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:15.669894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:15.670417 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:15.670633 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:15.670926 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 10:40:15.671247 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":135,"threads_started":1}
W20260924 10:40:15.672664 10330 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:40:15.672740 10330 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:40:15.672849 10327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:40:15.673602 10327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:40:15.679253 10327 catalog_manager.cc:1347] Generated new cluster ID: 77901132e60643bd826169bbf989c1d3
I20260924 10:40:15.679314 10327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:40:15.686818 10327 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:40:15.688054 10327 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:40:15.700225 10327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6: Generated new TSK 0
I20260924 10:40:15.700932 10327 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:40:15.737277 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:41783
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:15.870932 10336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:15.871201 10336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:15.871279 10336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:15.875375 10336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:15.875435 10336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:15.875495 10336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:40:15.880784 10336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:41783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:15.881880 10336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:15.882908 10336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:15.892506 10342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:15.892534 10341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:15.892627 10336 server_base.cc:1034] running on GCE node
W20260924 10:40:15.892531 10344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:15.893218 10336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:15.893920 10336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:15.895125 10336 hybrid_clock.cc:648] HybridClock initialized: now 1790246415895091 us; error 46 us; skew 500 ppm
Sep 24 10:40:15 dist-test-slave-j38h krb5kdc[10229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246415, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:15.898092 10336 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:40:15.898934 10336 webserver.cc:460] Webserver started at http://127.2.8.1:38139/ using document root <none> and password file <none>
I20260924 10:40:15.899475 10336 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:15.899564 10336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:15.899796 10336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:15.901859 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "444097686b6c4d14bb3301a564681f9d"
format_stamp: "Formatted at 2026-09-24 10:40:15 on dist-test-slave-j38h"
server_key: "3ea8cb6fbed721ce5473e8fe07397719"
server_key_iv: "8821488bbe661f4b9d45513f0788f2de"
server_key_version: "encryptionkey@0"
I20260924 10:40:15.902402 10336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "444097686b6c4d14bb3301a564681f9d"
format_stamp: "Formatted at 2026-09-24 10:40:15 on dist-test-slave-j38h"
server_key: "3ea8cb6fbed721ce5473e8fe07397719"
server_key_iv: "8821488bbe661f4b9d45513f0788f2de"
server_key_version: "encryptionkey@0"
I20260924 10:40:15.906018 10336 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:40:15.908485 10351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:15.909822 10336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:40:15.909973 10336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "444097686b6c4d14bb3301a564681f9d"
format_stamp: "Formatted at 2026-09-24 10:40:15 on dist-test-slave-j38h"
server_key: "3ea8cb6fbed721ce5473e8fe07397719"
server_key_iv: "8821488bbe661f4b9d45513f0788f2de"
server_key_version: "encryptionkey@0"
I20260924 10:40:15.910079 10336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:15.929593 10336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:15.932818 10336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:15.933063 10336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:15.933712 10336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:15.934597 10336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:15.934669 10336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:15.934741 10336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:15.934787 10336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:15.943660 10336 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:40651
I20260924 10:40:15.943679 10464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:40651 every 8 connection(s)
I20260924 10:40:15.944605 10336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
Sep 24 10:40:15 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246415, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:40:15.954555 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 10336
I20260924 10:40:15.954682 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:40:15.954981 2080 external_mini_cluster.cc:1428] Setting key 1482e14594fd0be47e59c2d42d135d33
I20260924 10:40:15.957765 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:15.946448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:59575 (local address 127.2.8.62:41783)
0924 10:40:15.946592 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:15.946596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:15.947592 (+ 996us) server_negotiation.cc:407] Connection header received
0924 10:40:15.948419 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:15.948422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:15.948517 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:15.948591 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:15.949730 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:15.950259 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:15.951344 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:15.951658 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:15.951762 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:15.954113 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:15.954116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:15.954118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:15.954140 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:15.955716 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:15.956238 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:15.956242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:15.956244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:15.956330 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:15.956668 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:15.956673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:15.956675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:15.956856 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:15.956960 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:15.957489 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 10:40:15.957608 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:40:15.958626 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:15.946821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:59575)
0924 10:40:15.947447 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:15.947487 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:15.948246 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:15.948752 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:15.948758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:15.949138 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:15.949601 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:15.949614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:15.950375 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:15.950378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:15.951236 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:15.951244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:15.951764 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:15.951811 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:15.951959 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:15.952509 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:15.952524 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:15.953960 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:15.955855 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:15.955860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:15.955876 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:15.956134 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:15.956427 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:15.956430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:15.956432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:15.956572 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:15.957004 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:15.957011 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:15.957246 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 10:40:15.957441 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 10:40:15.957772 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":108,"threads_started":1}
I20260924 10:40:15.959853 10465 heartbeater.cc:347] Connected to a master server at 127.2.8.62:41783
I20260924 10:40:15.960098 10465 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:15.960572 10465 heartbeater.cc:511] Master 127.2.8.62:41783 requested a full tablet report, sending...
I20260924 10:40:15.962100 10274 ts_manager.cc:194] Registered new tserver with Master: 444097686b6c4d14bb3301a564681f9d (127.2.8.1:40651)
I20260924 10:40:15.963192 10274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:59575
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:40:15.982290 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:41783
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:16.088868 10470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:16.089216 10470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:16.089301 10470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:16.093307 10470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:16.093384 10470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:16.093477 10470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:40:16.098572 10470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:41783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:16.099989 10470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:16.101054 10470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:16.110185 10475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:16.110297 10470 server_base.cc:1034] running on GCE node
W20260924 10:40:16.110219 10478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:16.110213 10476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:16.110852 10470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:16.111397 10470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:16.112563 10470 hybrid_clock.cc:648] HybridClock initialized: now 1790246416112542 us; error 30 us; skew 500 ppm
Sep 24 10:40:16 dist-test-slave-j38h krb5kdc[10229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246416, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:16.115384 10470 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:40:16.116197 10470 webserver.cc:460] Webserver started at http://127.2.8.2:37247/ using document root <none> and password file <none>
I20260924 10:40:16.116729 10470 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:16.116786 10470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:16.117077 10470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:16.119055 10470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "0745c11394c744ff955a0089c57df581"
format_stamp: "Formatted at 2026-09-24 10:40:16 on dist-test-slave-j38h"
server_key: "91406483f422e36eb96d6e21856ea0db"
server_key_iv: "00ce0b88786a3cc56a76b27aa967bf06"
server_key_version: "encryptionkey@0"
I20260924 10:40:16.119593 10470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "0745c11394c744ff955a0089c57df581"
format_stamp: "Formatted at 2026-09-24 10:40:16 on dist-test-slave-j38h"
server_key: "91406483f422e36eb96d6e21856ea0db"
server_key_iv: "00ce0b88786a3cc56a76b27aa967bf06"
server_key_version: "encryptionkey@0"
I20260924 10:40:16.123193 10470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:40:16.125627 10485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:16.126830 10470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:40:16.126979 10470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "0745c11394c744ff955a0089c57df581"
format_stamp: "Formatted at 2026-09-24 10:40:16 on dist-test-slave-j38h"
server_key: "91406483f422e36eb96d6e21856ea0db"
server_key_iv: "00ce0b88786a3cc56a76b27aa967bf06"
server_key_version: "encryptionkey@0"
I20260924 10:40:16.127086 10470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:16.143862 10470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:16.146597 10470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:16.146788 10470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:16.147341 10470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:16.148342 10470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:16.148411 10470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:16.148475 10470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:16.148526 10470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:16.157138 10470 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:43839
I20260924 10:40:16.157182 10598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:43839 every 8 connection(s)
I20260924 10:40:16.158056 10470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
Sep 24 10:40:16 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246416, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:40:16.168176 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 10470
I20260924 10:40:16.168327 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:40:16.168707 2080 external_mini_cluster.cc:1428] Setting key bb6a4ea9de08c9449347440baf448af1
I20260924 10:40:16.173025 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:16.159829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:38935 (local address 127.2.8.62:41783)
0924 10:40:16.159960 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:16.159968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:16.161054 (+ 1086us) server_negotiation.cc:407] Connection header received
0924 10:40:16.161891 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:16.161894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:16.161940 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:16.162015 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:16.163136 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.163657 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.164709 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.164920 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.165055 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:16.167438 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:16.167441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:16.167443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:16.167483 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:16.169195 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:16.169671 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:16.169674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:16.169677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:16.169779 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:16.171847 (+ 2068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:16.171850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:16.171852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:16.172040 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:16.172156 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:16.172707 (+ 551us) server_negotiation.cc:299] Negotiation successful
0924 10:40:16.172835 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:40:16.173875 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:16.160215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:38935)
0924 10:40:16.160849 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:16.160888 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:16.161713 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:16.162172 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:16.162178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:16.162551 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:16.163015 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:16.163027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.163758 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.163760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:16.164611 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:16.164619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.165047 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.165100 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:16.165241 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:16.165811 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:16.165826 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:16.167281 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:16.169323 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:16.169328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:16.169341 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:16.169588 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:16.169884 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:16.169887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:16.169889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:16.170167 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:16.172173 (+ 2006us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:16.172181 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:16.172457 (+ 276us) client_negotiation.cc:769] Sending connection context
0924 10:40:16.172654 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 10:40:16.173041 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":108,"threads_started":1}
I20260924 10:40:16.175071 10599 heartbeater.cc:347] Connected to a master server at 127.2.8.62:41783
I20260924 10:40:16.175339 10599 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:16.175983 10599 heartbeater.cc:511] Master 127.2.8.62:41783 requested a full tablet report, sending...
I20260924 10:40:16.177166 10274 ts_manager.cc:194] Registered new tserver with Master: 0745c11394c744ff955a0089c57df581 (127.2.8.2:43839)
I20260924 10:40:16.177743 10274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:38935
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:40:16.196321 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:41783
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:16.305135 10604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:16.305374 10604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:16.305415 10604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:16.309604 10604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:16.309656 10604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:16.309711 10604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:40:16.314819 10604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:41783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:16.315721 10604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:16.316620 10604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:16.325804 10610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:16.325804 10609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:16.325830 10612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:16.326130 10604 server_base.cc:1034] running on GCE node
I20260924 10:40:16.326479 10604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:16.327095 10604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:16.328255 10604 hybrid_clock.cc:648] HybridClock initialized: now 1790246416328244 us; error 23 us; skew 500 ppm
Sep 24 10:40:16 dist-test-slave-j38h krb5kdc[10229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246416, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:16.330890 10604 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:40:16.331753 10604 webserver.cc:460] Webserver started at http://127.2.8.3:45523/ using document root <none> and password file <none>
I20260924 10:40:16.332274 10604 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:16.332330 10604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:16.332484 10604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:16.334448 10604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "7e9e4892325e40e3a5a6795397780d4b"
format_stamp: "Formatted at 2026-09-24 10:40:16 on dist-test-slave-j38h"
server_key: "2726578c4c848b8576ed4445224d74d9"
server_key_iv: "287e6801c6e9cd14e7275ab807053501"
server_key_version: "encryptionkey@0"
I20260924 10:40:16.334949 10604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "7e9e4892325e40e3a5a6795397780d4b"
format_stamp: "Formatted at 2026-09-24 10:40:16 on dist-test-slave-j38h"
server_key: "2726578c4c848b8576ed4445224d74d9"
server_key_iv: "287e6801c6e9cd14e7275ab807053501"
server_key_version: "encryptionkey@0"
I20260924 10:40:16.338563 10604 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 10:40:16.340847 10619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:16.342109 10604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:40:16.342206 10604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "7e9e4892325e40e3a5a6795397780d4b"
format_stamp: "Formatted at 2026-09-24 10:40:16 on dist-test-slave-j38h"
server_key: "2726578c4c848b8576ed4445224d74d9"
server_key_iv: "287e6801c6e9cd14e7275ab807053501"
server_key_version: "encryptionkey@0"
I20260924 10:40:16.342285 10604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:16.357627 10604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:16.360358 10604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:16.360509 10604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:16.361081 10604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:16.361846 10604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:16.361891 10604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:16.361928 10604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:16.361944 10604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:16.370751 10604 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:38851
I20260924 10:40:16.370800 10732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:38851 every 8 connection(s)
I20260924 10:40:16.371660 10604 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:40:16.372372 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 10604
I20260924 10:40:16.372510 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:40:16.372854 2080 external_mini_cluster.cc:1428] Setting key 0d0c7da666aea1af5cc76e6f08675ef3
Sep 24 10:40:16 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246416, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:40:16.384920 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:16.373769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:44565 (local address 127.2.8.62:41783)
0924 10:40:16.373901 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:16.373906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:16.375267 (+ 1361us) server_negotiation.cc:407] Connection header received
0924 10:40:16.376202 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:16.376205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:16.376247 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:16.376315 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:16.377543 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.378053 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.379293 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.379493 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.379566 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:16.382025 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:16.382028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:16.382030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:16.382052 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:16.383243 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:16.383697 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:16.383700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:16.383701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:16.383767 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:16.384042 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:16.384044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:16.384045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:16.384206 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:16.384279 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:16.384715 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 10:40:16.384803 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:40:16.385838 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:16.374296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:44565)
0924 10:40:16.375105 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:16.375164 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:16.376018 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:16.376490 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:16.376496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:16.376874 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:16.377424 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:16.377438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.378160 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.378162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:16.379186 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:16.379196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.379601 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.379644 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:16.379789 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:16.380423 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:16.380440 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:16.381866 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:16.383342 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:16.383346 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:16.383356 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:16.383615 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:16.383841 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:16.383843 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:16.383844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:16.383967 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:16.384290 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:16.384294 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:16.384528 (+ 234us) client_negotiation.cc:769] Sending connection context
0924 10:40:16.384675 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 10:40:16.384899 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":166,"threads_started":1}
I20260924 10:40:16.387032 10733 heartbeater.cc:347] Connected to a master server at 127.2.8.62:41783
I20260924 10:40:16.387272 10733 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:16.387725 10733 heartbeater.cc:511] Master 127.2.8.62:41783 requested a full tablet report, sending...
I20260924 10:40:16.388648 10274 ts_manager.cc:194] Registered new tserver with Master: 7e9e4892325e40e3a5a6795397780d4b (127.2.8.3:38851)
I20260924 10:40:16.389274 10274 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:44565
I20260924 10:40:16.398540 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.1.keytab.
Entry for principal oryx/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.1.keytab.
WARNING: no policy specified for oryx/127.2.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.2.keytab.
Entry for principal oryx/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.2.keytab.
WARNING: no policy specified for oryx/127.2.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.3.keytab.
Entry for principal oryx/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.3.keytab.
I20260924 10:40:16.471259 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 10336
I20260924 10:40:16.476420 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 10470
I20260924 10:40:16.482021 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 10604
I20260924 10:40:16.487162 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 10242
I20260924 10:40:16.493233 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:41783
--webserver_interface=127.2.8.62
--webserver_port=45677
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:41783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:16.603986 10742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:16.604286 10742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:16.604359 10742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:16.608309 10742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:40:16.608407 10742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:16.608444 10742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:16.608481 10742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:40:16.608533 10742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:40:16.613659 10742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:41783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:41783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=45677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:16.614738 10742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:16.615775 10742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:16.624053 10748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:16.624081 10747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:16.624053 10750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:16.624336 10742 server_base.cc:1034] running on GCE node
I20260924 10:40:16.624954 10742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:16.626073 10742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:16.627254 10742 hybrid_clock.cc:648] HybridClock initialized: now 1790246416627193 us; error 87 us; skew 500 ppm
Sep 24 10:40:16 dist-test-slave-j38h krb5kdc[10229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246416, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:16.630069 10742 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:40:16.630972 10742 webserver.cc:460] Webserver started at http://127.2.8.62:45677/ using document root <none> and password file <none>
I20260924 10:40:16.631531 10742 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:16.631642 10742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:16.635387 10742 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 10:40:16.637555 10757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:16.638849 10742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:40:16.638981 10742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "4439ac941a4e4b7ea031d24cdeb383e6"
format_stamp: "Formatted at 2026-09-24 10:40:15 on dist-test-slave-j38h"
server_key: "40e03079b4ca5975281112f4e04e0e9e"
server_key_iv: "4c389e1a50aedf9b71ee0bd33955cee5"
server_key_version: "encryptionkey@0"
I20260924 10:40:16.639513 10742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:16.663450 10742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:16.666930 10742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:16.667143 10742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:16.674680 10742 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:41783
I20260924 10:40:16.674695 10809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:41783 every 8 connection(s)
I20260924 10:40:16.675745 10742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:40:16.679173 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 10742
I20260924 10:40:16.679625 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:40651
--local_ip_for_outbound_sockets=127.2.8.1
--tserver_master_addrs=127.2.8.62:41783
--webserver_port=38139
--webserver_interface=127.2.8.1
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.8.1
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:40:16.680166 10810 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:40:16.681788 10810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6: Bootstrap starting.
I20260924 10:40:16.704852 10810 log.cc:824] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:16.712152 10810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:40:16.712546 10810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6: Bootstrap complete.
I20260924 10:40:16.716686 10810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ac941a4e4b7ea031d24cdeb383e6" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41783 } }
I20260924 10:40:16.717139 10810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4439ac941a4e4b7ea031d24cdeb383e6, State: Initialized, Role: FOLLOWER
I20260924 10:40:16.717624 10810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ac941a4e4b7ea031d24cdeb383e6" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41783 } }
I20260924 10:40:16.717732 10810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:40:16.717819 10810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:40:16.717921 10810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:40:16.719797 10810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ac941a4e4b7ea031d24cdeb383e6" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41783 } }
I20260924 10:40:16.720140 10810 leader_election.cc:302] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4439ac941a4e4b7ea031d24cdeb383e6; no voters:
I20260924 10:40:16.720468 10810 leader_election.cc:288] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:40:16.720568 10814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:40:16.720769 10814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [term 2 LEADER]: Becoming Leader. State: Replica: 4439ac941a4e4b7ea031d24cdeb383e6, State: Running, Role: LEADER
I20260924 10:40:16.721077 10814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ac941a4e4b7ea031d24cdeb383e6" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41783 } }
I20260924 10:40:16.721688 10810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:40:16.722033 10814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4439ac941a4e4b7ea031d24cdeb383e6. Latest consensus state: current_term: 2 leader_uuid: "4439ac941a4e4b7ea031d24cdeb383e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ac941a4e4b7ea031d24cdeb383e6" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41783 } } }
I20260924 10:40:16.722146 10814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:16.722538 10815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4439ac941a4e4b7ea031d24cdeb383e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ac941a4e4b7ea031d24cdeb383e6" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 41783 } } }
I20260924 10:40:16.722693 10815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:16.723675 10818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:40:16.726912 10818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:40:16.727511 10818 catalog_manager.cc:1259] Loaded cluster ID: 77901132e60643bd826169bbf989c1d3
I20260924 10:40:16.727569 10818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:40:16.729928 10818 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:40:16.730561 10818 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4439ac941a4e4b7ea031d24cdeb383e6: Loaded TSK: 0
I20260924 10:40:16.731747 10818 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:40:16.804665 10812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:16.804908 10812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:16.804947 10812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:16.808923 10812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:16.808997 10812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:16.809118 10812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:40:16.814353 10812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.1.keytab
--principal=oryx/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:40651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=38139
--tserver_master_addrs=127.2.8.62:41783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:16.815330 10812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:16.816300 10812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:16.825985 10836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:16.825994 10839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:16.826150 10837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:16.826399 10812 server_base.cc:1034] running on GCE node
I20260924 10:40:16.826881 10812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:16.827535 10812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:16.828719 10812 hybrid_clock.cc:648] HybridClock initialized: now 1790246416828694 us; error 45 us; skew 500 ppm
Sep 24 10:40:16 dist-test-slave-j38h krb5kdc[10229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246416, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:16.831588 10812 init.cc:377] Logged in from keytab as oryx/127.2.8.1@KRBTEST.COM (short username oryx)
I20260924 10:40:16.832494 10812 webserver.cc:460] Webserver started at http://127.2.8.1:38139/ using document root <none> and password file <none>
I20260924 10:40:16.833117 10812 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:16.833194 10812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:16.837234 10812 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:40:16.839454 10846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:16.840762 10812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:40:16.840904 10812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "444097686b6c4d14bb3301a564681f9d"
format_stamp: "Formatted at 2026-09-24 10:40:15 on dist-test-slave-j38h"
server_key: "3ea8cb6fbed721ce5473e8fe07397719"
server_key_iv: "8821488bbe661f4b9d45513f0788f2de"
server_key_version: "encryptionkey@0"
I20260924 10:40:16.841501 10812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:16.854430 10812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:16.859808 10812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:16.860055 10812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:16.860683 10812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:16.861680 10812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:16.861749 10812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:16.861814 10812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:16.861840 10812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:16.871429 10812 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:40651
I20260924 10:40:16.871496 10959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:40651 every 8 connection(s)
I20260924 10:40:16.872463 10812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:40:16.876760 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 10812
I20260924 10:40:16.877278 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:43839
--local_ip_for_outbound_sockets=127.2.8.2
--tserver_master_addrs=127.2.8.62:41783
--webserver_port=37247
--webserver_interface=127.2.8.2
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.8.2
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:40:16 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:16 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:16.888327 10963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:16.874910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:43953)
0924 10:40:16.875666 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:16.875705 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:16.876512 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:16.877932 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:16.877942 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:16.878613 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:16.879367 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:16.879390 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.880768 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.880783 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:16.882619 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:16.882638 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.883386 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.883462 (+ 76us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:16.883725 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:16.884454 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:16.884486 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:16.887338 (+ 2852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":116,"threads_started":1}
W20260924 10:40:16.888320 10962 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:16.874795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:43953 (local address 127.2.8.62:41783)
0924 10:40:16.875361 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:16.875382 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:16.875838 (+ 456us) server_negotiation.cc:407] Connection header received
0924 10:40:16.876960 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:16.876998 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:16.877340 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:16.877870 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:16.879571 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.880595 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.882774 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.883252 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.883378 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:16.887573 (+ 4195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:43953: BlockingRecv error: recv got EOF from 127.2.8.1:43953 (error 108)
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":146,"threads_started":1}
W20260924 10:40:16.889143 10960 heartbeater.cc:644] Failed to heartbeat to 127.2.8.62:41783 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.8.62:41783: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Sep 24 10:40:16 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:16 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:16.894781 10963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:16.889657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:34361)
0924 10:40:16.889776 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:16.889793 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:16.889893 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:16.890178 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:16.890181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:16.890344 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:16.890430 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:16.890435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.891154 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.891160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:16.891768 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:16.891776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.892203 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.892221 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:16.892309 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:16.892817 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:16.892837 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:16.894636 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 10:40:16.894973 10962 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:16.889830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:34361 (local address 127.2.8.62:41783)
0924 10:40:16.889980 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:16.889984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:16.890003 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:40:16.890036 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:16.890038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:16.890077 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:16.890163 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:16.890544 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.891049 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.891884 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.892107 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.892215 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:16.894759 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:34361: BlockingRecv error: recv got EOF from 127.2.8.1:34361 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 10:40:16 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:16 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:16.900574 10963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:16.895458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:58057)
0924 10:40:16.895646 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:16.895660 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:16.895751 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:16.896018 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:16.896021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:16.896169 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:16.896232 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:16.896237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.896922 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.896928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:16.897500 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:16.897509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.897929 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.897944 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:16.898024 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:16.898574 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:16.898585 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:16.900452 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118}
W20260924 10:40:16.900723 10962 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:16.895567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:58057 (local address 127.2.8.62:41783)
0924 10:40:16.895668 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:16.895674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:16.895706 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:40:16.895855 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:16.895860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:16.895903 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:16.896014 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:16.896342 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.896818 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.897620 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:16.897833 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:16.897941 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:16.900599 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:58057: BlockingRecv error: recv got EOF from 127.2.8.1:58057 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 10:40:16.901134 10960 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:40:17.000953 10964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:17.001294 10964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:17.001343 10964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:17.005426 10964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:17.005499 10964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:17.005563 10964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:40:17.010972 10964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.2.keytab
--principal=oryx/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:43839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=37247
--tserver_master_addrs=127.2.8.62:41783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:17.012069 10964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:17.013088 10964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:17.022817 10972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:17.022805 10970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:17.022840 10969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:17.023311 10964 server_base.cc:1034] running on GCE node
I20260924 10:40:17.023722 10964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:17.024401 10964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:17.025579 10964 hybrid_clock.cc:648] HybridClock initialized: now 1790246417025571 us; error 33 us; skew 500 ppm
Sep 24 10:40:17 dist-test-slave-j38h krb5kdc[10229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246417, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:17.028263 10964 init.cc:377] Logged in from keytab as oryx/127.2.8.2@KRBTEST.COM (short username oryx)
I20260924 10:40:17.029171 10964 webserver.cc:460] Webserver started at http://127.2.8.2:37247/ using document root <none> and password file <none>
I20260924 10:40:17.029723 10964 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:17.029829 10964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:17.033493 10964 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 10:40:17.035467 10979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:17.036650 10964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:40:17.036778 10964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "0745c11394c744ff955a0089c57df581"
format_stamp: "Formatted at 2026-09-24 10:40:16 on dist-test-slave-j38h"
server_key: "91406483f422e36eb96d6e21856ea0db"
server_key_iv: "00ce0b88786a3cc56a76b27aa967bf06"
server_key_version: "encryptionkey@0"
I20260924 10:40:17.037370 10964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:17.049935 10964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:17.053194 10964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:17.053401 10964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:17.053998 10964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:17.054806 10964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:17.054876 10964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:17.054940 10964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:17.054986 10964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:17.063678 10964 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:43839
I20260924 10:40:17.063702 11092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:43839 every 8 connection(s)
I20260924 10:40:17.064697 10964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
Sep 24 10:40:17 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:17 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:17.074674 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 10964
I20260924 10:40:17.075160 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:38851
--local_ip_for_outbound_sockets=127.2.8.3
--tserver_master_addrs=127.2.8.62:41783
--webserver_port=45523
--webserver_interface=127.2.8.3
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.8.3
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:17.075779 10962 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:17.066675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:53859 (local address 127.2.8.62:41783)
0924 10:40:17.066842 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:17.066847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:17.067943 (+ 1096us) server_negotiation.cc:407] Connection header received
0924 10:40:17.068811 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:17.068813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:17.068868 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:17.068944 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:17.070236 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.070788 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.071894 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.072101 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.072179 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:17.075657 (+ 3478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:53859: BlockingRecv error: recv got EOF from 127.2.8.2:53859 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:40:17.076516 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:17.067132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:53859)
0924 10:40:17.067791 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:17.067832 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:17.068625 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:17.069185 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:17.069192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:17.069576 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:17.070053 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:17.070066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.070905 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.070907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:17.071776 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:17.071785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.072188 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.072233 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:17.072369 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:17.072941 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:17.072958 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:17.075533 (+ 2575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":123,"threads_started":1}
W20260924 10:40:17.077065 11093 heartbeater.cc:644] Failed to heartbeat to 127.2.8.62:41783 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.8.62:41783: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Sep 24 10:40:17 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:17 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:17.082695 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:17.077533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:56749)
0924 10:40:17.077627 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:17.077641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:17.077717 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:17.077953 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:17.077956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:17.078122 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:17.078205 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:17.078210 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.078929 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.078934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:17.079479 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:17.079486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.079921 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.079936 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:17.080025 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:17.080571 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:17.080582 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:17.082549 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 10:40:17.082780 10962 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:17.077623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:56749 (local address 127.2.8.62:41783)
0924 10:40:17.077727 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:17.077731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:17.077746 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:40:17.077801 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:17.077804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:17.077849 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:17.077939 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:17.078318 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.078824 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.079593 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.079824 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.079931 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:17.082664 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:56749: BlockingRecv error: recv got EOF from 127.2.8.2:56749 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 10:40:17 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:17 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:17.088053 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:17.083215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:56063)
0924 10:40:17.083318 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:17.083335 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:17.083426 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:17.083641 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:17.083643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:17.083782 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:17.083847 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:17.083851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.084530 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.084536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:17.085085 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:17.085093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.085503 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.085518 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:17.085592 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:17.086107 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:17.086118 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:17.087922 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 10:40:17.088150 10962 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:17.083316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:56063 (local address 127.2.8.62:41783)
0924 10:40:17.083439 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:17.083444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:17.083462 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:40:17.083500 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:17.083502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:17.083546 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:17.083627 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:17.083949 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.084426 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.085192 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.085409 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.085513 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:17.088025 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:56063: BlockingRecv error: recv got EOF from 127.2.8.2:56063 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 10:40:17.088344 11093 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:40:17.187664 11096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:17.187976 11096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:17.188060 11096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:17.192184 11096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:17.192269 11096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:17.192363 11096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:40:17.197578 11096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/krb5kdc/oryx_127.2.8.3.keytab
--principal=oryx/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:38851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=45523
--tserver_master_addrs=127.2.8.62:41783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:17.198670 11096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:17.199714 11096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:40:17.209208 11096 server_base.cc:1034] running on GCE node
W20260924 10:40:17.209231 11102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:17.209246 11101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:17.209197 11104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:17.209911 11096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:17.210553 11096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:17.211755 11096 hybrid_clock.cc:648] HybridClock initialized: now 1790246417211741 us; error 24 us; skew 500 ppm
Sep 24 10:40:17 dist-test-slave-j38h krb5kdc[10229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246417, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:17.214617 11096 init.cc:377] Logged in from keytab as oryx/127.2.8.3@KRBTEST.COM (short username oryx)
I20260924 10:40:17.215447 11096 webserver.cc:460] Webserver started at http://127.2.8.3:45523/ using document root <none> and password file <none>
I20260924 10:40:17.216008 11096 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:17.216109 11096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:17.220203 11096 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:40:17.222718 11111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:17.224313 11096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:40:17.224406 11096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "7e9e4892325e40e3a5a6795397780d4b"
format_stamp: "Formatted at 2026-09-24 10:40:16 on dist-test-slave-j38h"
server_key: "2726578c4c848b8576ed4445224d74d9"
server_key_iv: "287e6801c6e9cd14e7275ab807053501"
server_key_version: "encryptionkey@0"
I20260924 10:40:17.224907 11096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:17.238612 11096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:17.242012 11096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:17.242161 11096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:17.242738 11096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:17.243593 11096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:17.243667 11096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:17.243728 11096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:17.243762 11096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:17.252488 11096 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:38851
I20260924 10:40:17.252553 11224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:38851 every 8 connection(s)
I20260924 10:40:17.253531 11096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:40:17.261982 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 11096
Sep 24 10:40:17 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:17 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:40:17.264096 10962 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:17.255446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:48911 (local address 127.2.8.62:41783)
0924 10:40:17.255568 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:17.255579 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:17.256537 (+ 958us) server_negotiation.cc:407] Connection header received
0924 10:40:17.257352 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:17.257355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:17.257401 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:17.257501 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:17.258599 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.259106 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.260245 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.260445 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.260544 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:17.264001 (+ 3457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:48911: BlockingRecv error: recv got EOF from 127.2.8.3:48911 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:40:17.264693 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:17.255729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:48911)
0924 10:40:17.256385 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:17.256426 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:17.257175 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:17.257618 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:17.257625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:17.257949 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:17.258464 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:17.258479 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.259212 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.259215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:17.260143 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:17.260152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.260553 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.260599 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:17.260731 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:17.261299 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:17.261320 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:17.263888 (+ 2568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":116,"threads_started":1}
W20260924 10:40:17.265161 11225 heartbeater.cc:644] Failed to heartbeat to 127.2.8.62:41783 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.8.62:41783: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Sep 24 10:40:17 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:17 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:17.270569 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:17.265559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:49851)
0924 10:40:17.265642 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:17.265656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:17.265724 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:17.265877 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:17.265879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:17.265994 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:17.266071 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:17.266075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.266795 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.266799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:17.267566 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:17.267574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.268058 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.268071 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:17.268155 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:17.268515 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:17.268532 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:17.270412 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 10:40:17.270692 10962 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:17.265627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:49851 (local address 127.2.8.62:41783)
0924 10:40:17.265726 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:17.265729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:17.265740 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:40:17.265773 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:17.265775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:17.265806 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:17.265862 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:17.266171 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.266679 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.267668 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.267935 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.268059 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:17.270550 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:49851: BlockingRecv error: recv got EOF from 127.2.8.3:49851 (error 108)
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":28}
I20260924 10:40:17.274107 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:17.263048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.2.8.62:41783)
0924 10:40:17.263293 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:17.263297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:17.263319 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:40:17.264334 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:17.264337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:17.264389 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:17.264491 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:17.265323 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.266224 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.267371 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.267583 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.267718 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:17.269411 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:17.269417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:17.269434 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:17.269483 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:17.271822 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:17.272267 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:17.272274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:17.272281 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:17.272409 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:17.272692 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:17.272697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:17.272699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:17.273127 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:17.273340 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:17.273611 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 10:40:17.273878 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":167,"thread_start_us":98,"threads_started":1}
Sep 24 10:40:17 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:17 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:17.275797 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:17.271013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:49709)
0924 10:40:17.271144 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:17.271175 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:17.271251 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:17.271479 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:17.271482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:17.271634 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:17.271698 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:17.271703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.272396 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.272401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:17.272922 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:17.272929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.273395 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.273411 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:17.273498 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:17.273948 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:17.273962 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:17.275688 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 10:40:17.275899 10962 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:17.271109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:49709 (local address 127.2.8.62:41783)
0924 10:40:17.271257 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:17.271261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:17.271275 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:40:17.271335 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:17.271338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:17.271372 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:17.271460 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:17.271808 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.272293 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.273078 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.273303 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.273407 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:17.275794 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:49709: BlockingRecv error: recv got EOF from 127.2.8.3:49709 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260924 10:40:17.276034 11225 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 10:40:17 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:17 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:17.907779 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:17.901771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:43025)
0924 10:40:17.902053 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:17.902066 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:17.902152 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:17.902420 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:17.902422 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:17.902597 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:17.902687 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:17.902692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.903490 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.903492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:17.904055 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:17.904063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.904506 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.904520 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:17.904577 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:17.905182 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:17.905194 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:17.907585 (+ 2391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":101,"threads_started":1}
W20260924 10:40:17.907855 11231 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:17.901936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:43025 (local address 127.2.8.62:41783)
0924 10:40:17.902197 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:17.902201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:17.902213 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:40:17.902252 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:17.902254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:17.902297 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:17.902402 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:17.902790 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.903376 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.904176 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:17.904403 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:17.904491 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:17.907711 (+ 3220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:43025: BlockingRecv error: recv got EOF from 127.2.8.1:43025 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":113,"threads_started":1}
Sep 24 10:40:18 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:18 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:18.094615 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:18.089032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:38015)
0924 10:40:18.089323 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:18.089337 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:18.089433 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:18.089679 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:18.089681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:18.089851 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:18.089966 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:18.089972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:18.090796 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:18.090798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:18.091423 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:18.091433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:18.091905 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:18.091922 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:18.092020 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:18.092576 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:18.092597 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:18.094484 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":99,"threads_started":1}
W20260924 10:40:18.094770 11231 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:18.089087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:38015 (local address 127.2.8.62:41783)
0924 10:40:18.089225 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:18.089229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:18.089442 (+ 213us) server_negotiation.cc:407] Connection header received
0924 10:40:18.089525 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:18.089528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:18.089570 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:18.089644 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:18.090100 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:18.090698 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:18.091576 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:18.091801 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:18.091911 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:18.094619 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:38015: BlockingRecv error: recv got EOF from 127.2.8.2:38015 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 10:40:18 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:18 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:18.282784 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:18.276734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:53887)
0924 10:40:18.277107 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:18.277124 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:18.277291 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:18.277669 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:18.277672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:18.277892 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:18.277989 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:18.277995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:18.278826 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:18.278831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:18.279479 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:18.279487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:18.279963 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:18.279981 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:18.280093 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:18.280596 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:18.280608 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:18.282587 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":100,"threads_started":1}
W20260924 10:40:18.282836 11231 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:18.276896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:53887 (local address 127.2.8.62:41783)
0924 10:40:18.277107 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:18.277113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:18.277227 (+ 114us) server_negotiation.cc:407] Connection header received
0924 10:40:18.277468 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:18.277473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:18.277532 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:18.277624 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:18.278114 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:18.278706 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:18.279605 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:18.279859 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:18.279979 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:18.282710 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:53887: BlockingRecv error: recv got EOF from 127.2.8.3:53887 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 24 10:40:18 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:18 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:18.913727 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:18.908509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:45201)
0924 10:40:18.908777 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:18.908800 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:18.908894 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:18.909159 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:18.909160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:18.909331 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:18.909424 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:18.909430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:18.910135 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:18.910137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:18.910674 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:18.910681 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:18.911061 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:18.911075 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:18.911134 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:18.911710 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:18.911722 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:18.913617 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":109,"threads_started":1}
W20260924 10:40:18.913878 11235 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:18.908577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:45201 (local address 127.2.8.62:41783)
0924 10:40:18.908800 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:18.908803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:18.908849 (+ 46us) server_negotiation.cc:407] Connection header received
0924 10:40:18.908997 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:18.909000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:18.909043 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:18.909119 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:18.909520 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:18.910044 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:18.910765 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:18.910983 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:18.911055 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:18.913744 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:45201: BlockingRecv error: recv got EOF from 127.2.8.1:45201 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
Sep 24 10:40:19 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:19 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:19.100313 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:19.095359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:40339)
0924 10:40:19.095653 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:19.095665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:19.095758 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:19.095996 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:19.095998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:19.096164 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:19.096253 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:19.096258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:19.096953 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:19.096955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:19.097532 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:19.097539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:19.097936 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:19.097949 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:19.098007 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:19.098489 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:19.098501 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:19.100207 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":123,"threads_started":1}
W20260924 10:40:19.100469 11235 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:19.095418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:40339 (local address 127.2.8.62:41783)
0924 10:40:19.095562 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:19.095565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:19.095733 (+ 168us) server_negotiation.cc:407] Connection header received
0924 10:40:19.095837 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:19.095839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:19.095882 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:19.095950 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:19.096346 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:19.096854 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:19.097627 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:19.097838 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:19.097914 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:19.100341 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:40339: BlockingRecv error: recv got EOF from 127.2.8.2:40339 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 10:40:19 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:19 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:19.289163 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:19.283597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:50961)
0924 10:40:19.283957 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:19.283970 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:19.284064 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:19.284313 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:19.284316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:19.284527 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:19.284626 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:19.284632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:19.285386 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:19.285389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:19.285964 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:19.285972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:19.286402 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:19.286417 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:19.286478 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:19.286966 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:19.286977 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:19.289002 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":192,"threads_started":1}
W20260924 10:40:19.289320 11235 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:19.283754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:50961 (local address 127.2.8.62:41783)
0924 10:40:19.283884 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:19.283895 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:19.284036 (+ 141us) server_negotiation.cc:407] Connection header received
0924 10:40:19.284130 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:19.284132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:19.284177 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:19.284244 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:19.284733 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:19.285267 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:19.286064 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:19.286275 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:19.286355 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:19.289164 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:50961: BlockingRecv error: recv got EOF from 127.2.8.3:50961 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 10:40:19 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:19 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:19.920704 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:19.914763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:53181)
0924 10:40:19.915028 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:19.915042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:19.915177 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:19.915496 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:19.915498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:19.915669 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:19.915760 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:19.915765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:19.916925 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:19.916928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:19.917810 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:19.917820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:19.918241 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:19.918255 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:19.918312 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:19.918844 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:19.918855 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:19.920586 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
W20260924 10:40:19.920923 11239 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:19.914847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:53181 (local address 127.2.8.62:41783)
0924 10:40:19.915116 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:19.915120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:19.915136 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:40:19.915275 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:19.915278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:19.915322 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:19.915473 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:19.915908 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:19.916820 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:19.917909 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:19.918150 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:19.918235 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:19.920757 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:53181: BlockingRecv error: recv got EOF from 127.2.8.1:53181 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":105,"threads_started":1}
Sep 24 10:40:20 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:20 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:20.106726 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:20.101109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:40251)
0924 10:40:20.101355 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:20.101368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:20.101511 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:20.101806 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:20.101808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:20.101997 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:20.102100 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:20.102105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:20.102822 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:20.102824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:20.103465 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:20.103474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:20.103897 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:20.103913 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:20.104017 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:20.104577 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:20.104591 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:20.106596 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":92,"threads_started":1}
W20260924 10:40:20.106832 11239 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:20.101197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:40251 (local address 127.2.8.62:41783)
0924 10:40:20.101330 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:20.101334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:20.101437 (+ 103us) server_negotiation.cc:407] Connection header received
0924 10:40:20.101608 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:20.101611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:20.101669 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:20.101739 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:20.102202 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:20.102723 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:20.103584 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:20.103805 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:20.103910 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:20.106725 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:40251: BlockingRecv error: recv got EOF from 127.2.8.2:40251 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 10:40:20 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:20 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:20.295288 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:20.289980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:39813)
0924 10:40:20.290266 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:20.290300 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:20.290414 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:20.290714 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:20.290717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:20.290966 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:20.291092 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:20.291100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:20.291845 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:20.291847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:20.292403 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:20.292411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:20.292792 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:20.292807 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:20.292865 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:20.293371 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:20.293382 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:20.295169 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":100,"threads_started":1}
W20260924 10:40:20.295406 11239 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:20.289997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:39813 (local address 127.2.8.62:41783)
0924 10:40:20.290142 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:20.290149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:20.290379 (+ 230us) server_negotiation.cc:407] Connection header received
0924 10:40:20.290527 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:20.290532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:20.290591 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:20.290682 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:20.291207 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:20.291727 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:20.292511 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:20.292713 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:20.292787 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:20.295295 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:39813: BlockingRecv error: recv got EOF from 127.2.8.3:39813 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 10:40:20 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:20 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:20.935678 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:20.921636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:56769)
0924 10:40:20.921890 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:20.921903 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:20.922012 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:20.922249 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:20.922251 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:20.922511 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:20.922626 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:20.922633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:20.923341 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:20.923343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:20.923921 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:20.923929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:20.924338 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:20.924352 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:20.924429 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:20.933662 (+ 9233us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:20.933677 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:20.935551 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":84,"threads_started":1}
W20260924 10:40:20.935921 11243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:20.921686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:56769 (local address 127.2.8.62:41783)
0924 10:40:20.921914 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:20.921918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:20.921953 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:40:20.922090 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:20.922092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:20.922136 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:20.922216 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:20.922720 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:20.923250 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:20.924029 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:20.924256 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:20.924336 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:20.935777 (+ 11441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:56769: BlockingRecv error: recv got EOF from 127.2.8.1:56769 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
Sep 24 10:40:21 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:21 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:21.113749 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:21.107606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:45229)
0924 10:40:21.107917 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:21.107933 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:21.108082 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:21.108356 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:21.108359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:21.108674 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:21.108772 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:21.108778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:21.109625 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:21.109631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:21.110203 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:21.110210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:21.110668 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:21.110684 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:21.110768 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:21.111355 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:21.111425 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:21.113565 (+ 2140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":83,"threads_started":1}
W20260924 10:40:21.113819 11243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:21.107661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:45229 (local address 127.2.8.62:41783)
0924 10:40:21.107881 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:21.107885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:21.108030 (+ 145us) server_negotiation.cc:407] Connection header received
0924 10:40:21.108190 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:21.108194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:21.108248 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:21.108332 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:21.108922 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:21.109512 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:21.110326 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:21.110567 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:21.110682 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:21.113700 (+ 3018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:45229: BlockingRecv error: recv got EOF from 127.2.8.2:45229 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 24 10:40:21 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:21 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:21.301939 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:21.296188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:49107)
0924 10:40:21.296457 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:21.296471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:21.296565 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:21.296810 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:21.296812 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:21.297021 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:21.297115 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:21.297120 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:21.297885 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:21.297888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:21.298489 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:21.298499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:21.299087 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:21.299103 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:21.299224 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:21.299935 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:21.299952 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:21.301825 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
W20260924 10:40:21.302173 11243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:21.296329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:49107 (local address 127.2.8.62:41783)
0924 10:40:21.296541 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:21.296545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:21.296559 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:40:21.296653 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:21.296656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:21.296702 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:21.296771 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:21.297216 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:21.297756 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:21.298649 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:21.298947 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:21.299079 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:21.302009 (+ 2930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:49107: BlockingRecv error: recv got EOF from 127.2.8.3:49107 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Sep 24 10:40:21 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:21 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:21.942375 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:21.936557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:38937)
0924 10:40:21.936832 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:21.936844 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:21.936925 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:21.937210 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:21.937212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:21.937383 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:21.937480 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:21.937485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:21.938215 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:21.938218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:21.938765 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:21.938772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:21.939202 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:21.939217 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:21.939277 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:21.939847 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:21.939859 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:21.942175 (+ 2316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":97,"threads_started":1}
W20260924 10:40:21.942515 11247 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:21.936602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:38937 (local address 127.2.8.62:41783)
0924 10:40:21.936867 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:21.936870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:21.936887 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:40:21.937049 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:21.937051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:21.937092 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:21.937171 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:21.937578 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:21.938106 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:21.938886 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:21.939115 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:21.939191 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:21.942352 (+ 3161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:38937: BlockingRecv error: recv got EOF from 127.2.8.1:38937 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":117,"threads_started":1}
Sep 24 10:40:22 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:22 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:22.120028 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:22.114698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:53323)
0924 10:40:22.115003 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:22.115017 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:22.115099 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:22.115340 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:22.115342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:22.115523 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:22.115615 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:22.115620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:22.116376 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:22.116378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:22.116915 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:22.116921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:22.117395 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:22.117410 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:22.117471 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:22.117973 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:22.117997 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:22.119908 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":127,"threads_started":1}
W20260924 10:40:22.120198 11247 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:22.114974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:53323 (local address 127.2.8.62:41783)
0924 10:40:22.115127 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:22.115131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:22.115144 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:40:22.115182 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:22.115185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:22.115227 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:22.115313 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:22.115723 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:22.116276 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:22.117081 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:22.117306 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:22.117391 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:22.120054 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:53323: BlockingRecv error: recv got EOF from 127.2.8.2:53323 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 10:40:22 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:22 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:22.308051 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:22.302733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:41727)
0924 10:40:22.303038 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:22.303057 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:22.303146 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:22.303336 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:22.303338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:22.303562 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:22.303657 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:22.303663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:22.304400 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:22.304449 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:22.305096 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:22.305105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:22.305511 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:22.305527 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:22.305589 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:22.306072 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:22.306083 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:22.307911 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":129,"threads_started":1}
W20260924 10:40:22.308229 11247 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:22.302851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:41727 (local address 127.2.8.62:41783)
0924 10:40:22.302999 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:22.303003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:22.303106 (+ 103us) server_negotiation.cc:407] Connection header received
0924 10:40:22.303211 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:22.303213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:22.303255 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:22.303322 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:22.303763 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:22.304290 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:22.305204 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:22.305413 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:22.305502 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:22.308086 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:41727: BlockingRecv error: recv got EOF from 127.2.8.3:41727 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 10:40:22 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:22 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:22.959656 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:22.943745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:40023)
0924 10:40:22.944072 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:22.944096 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:22.944252 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:22.944719 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:22.944722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:22.944933 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:22.945112 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:22.945122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:22.946035 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:22.946042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:22.946613 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:22.946621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:22.947096 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:22.947113 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:22.947198 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:22.957312 (+ 10114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:22.957332 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:22.959528 (+ 2196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":138,"threads_started":1}
W20260924 10:40:22.959869 11251 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:22.943790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:40023 (local address 127.2.8.62:41783)
0924 10:40:22.944097 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:22.944102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:22.944215 (+ 113us) server_negotiation.cc:407] Connection header received
0924 10:40:22.944498 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:22.944501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:22.944558 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:22.944699 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:22.945220 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:22.945912 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:22.946743 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:22.946991 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:22.947108 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:22.959686 (+ 12578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:40023: BlockingRecv error: recv got EOF from 127.2.8.1:40023 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":113,"threads_started":1}
Sep 24 10:40:23 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:23 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:23.127488 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:23.120871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:43065)
0924 10:40:23.121143 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:23.121160 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:23.121252 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:23.121553 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:23.121555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:23.121727 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:23.121877 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:23.121884 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:23.122744 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:23.122747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:23.123441 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:23.123450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:23.123938 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:23.123952 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:23.124016 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:23.125489 (+ 1473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:23.125503 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:23.127356 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":141,"threads_started":1}
W20260924 10:40:23.127707 11251 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:23.121062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:43065 (local address 127.2.8.62:41783)
0924 10:40:23.121192 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:23.121196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:23.121210 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:40:23.121348 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:23.121352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:23.121406 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:23.121492 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:23.121985 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:23.122656 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:23.123575 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:23.123836 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:23.123938 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:23.127576 (+ 3638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:43065: BlockingRecv error: recv got EOF from 127.2.8.2:43065 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 10:40:23 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:23 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:23.315568 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:23.308941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:43845)
0924 10:40:23.309333 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:23.309348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:23.309541 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:23.309833 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:23.309836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:23.310047 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:23.310135 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:23.310141 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:23.310917 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:23.310924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:23.311497 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:23.311504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:23.311980 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:23.311998 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:23.312085 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:23.313606 (+ 1521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:23.313618 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:23.315446 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":104,"threads_started":1}
W20260924 10:40:23.315757 11251 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:23.309106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:43845 (local address 127.2.8.62:41783)
0924 10:40:23.309291 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:23.309296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:23.309437 (+ 141us) server_negotiation.cc:407] Connection header received
0924 10:40:23.309651 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:23.309654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:23.309704 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:23.309814 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:23.310258 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:23.310800 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:23.311627 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:23.311868 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:23.311991 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:23.315587 (+ 3596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:43845: BlockingRecv error: recv got EOF from 127.2.8.3:43845 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 10:40:23 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:23 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:23.967494 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:23.960595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:43023)
0924 10:40:23.960963 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:23.961034 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:23.961158 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:23.961508 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:23.961512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:23.961773 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:23.961922 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:23.961930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:23.963045 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:23.963049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:23.963596 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:23.963604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:23.964006 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:23.964022 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:23.964081 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:23.965520 (+ 1439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:23.965533 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:23.967375 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":104,"threads_started":1}
W20260924 10:40:23.967679 11255 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:23.960671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:43023 (local address 127.2.8.62:41783)
0924 10:40:23.960908 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:23.960929 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:23.961151 (+ 222us) server_negotiation.cc:407] Connection header received
0924 10:40:23.961255 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:23.961258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:23.961303 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:23.961435 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:23.962051 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:23.962905 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:23.963700 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:23.963912 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:23.963988 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:23.967532 (+ 3544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:43023: BlockingRecv error: recv got EOF from 127.2.8.1:43023 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
Sep 24 10:40:24 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:24 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:24.134425 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:24.128241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:55503)
0924 10:40:24.128652 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:24.128665 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:24.128795 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:24.129113 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:24.129117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:24.129318 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:24.129406 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:24.129412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:24.130241 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:24.130246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:24.130816 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:24.130824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:24.131278 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:24.131295 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:24.131380 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:24.132419 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:24.132437 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:24.134309 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":107,"threads_started":1}
W20260924 10:40:24.134620 11255 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:24.128430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:55503 (local address 127.2.8.62:41783)
0924 10:40:24.128569 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:24.128573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:24.128742 (+ 169us) server_negotiation.cc:407] Connection header received
0924 10:40:24.128894 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:24.128896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:24.128951 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:24.129074 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:24.129571 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:24.130124 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:24.130949 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:24.131175 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:24.131291 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:24.134457 (+ 3166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:55503: BlockingRecv error: recv got EOF from 127.2.8.2:55503 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 10:40:24 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:24 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:24.322080 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:24.316355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:43055)
0924 10:40:24.316693 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:24.316706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:24.316800 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:24.317037 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:24.317039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:24.317210 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:24.317299 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:24.317304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:24.318004 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:24.318006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:24.318577 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:24.318585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:24.319011 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:24.319027 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:24.319092 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:24.320117 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:24.320128 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:24.321967 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":86,"threads_started":1}
W20260924 10:40:24.322273 11255 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:24.316429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:43055 (local address 127.2.8.62:41783)
0924 10:40:24.316537 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:24.316541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:24.316778 (+ 237us) server_negotiation.cc:407] Connection header received
0924 10:40:24.316868 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:24.316871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:24.316915 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:24.317023 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:24.317399 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:24.317901 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:24.318682 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:24.318888 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:24.318963 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:24.322109 (+ 3146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:43055: BlockingRecv error: recv got EOF from 127.2.8.3:43055 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 10:40:24 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:24 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:24.974808 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:24.968390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:56211)
0924 10:40:24.968709 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:24.968722 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:24.968810 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:24.969159 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:24.969161 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:24.969333 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:24.969430 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:24.969435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:24.970307 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:24.970309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:24.970847 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:24.970854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:24.971307 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:24.971321 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:24.971379 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:24.972743 (+ 1364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:24.972755 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:24.974651 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":91,"threads_started":1}
W20260924 10:40:24.974993 11259 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:24.968696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:56211 (local address 127.2.8.62:41783)
0924 10:40:24.968917 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:24.968920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:24.968932 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:40:24.969000 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:24.969002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:24.969043 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:24.969116 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:24.969585 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:24.970205 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:24.970968 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:24.971215 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:24.971302 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:24.974818 (+ 3516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:56211: BlockingRecv error: recv got EOF from 127.2.8.1:56211 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
Sep 24 10:40:25 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:25 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:25.141815 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:25.135179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:39005)
0924 10:40:25.135444 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:25.135457 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:25.135563 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:25.135809 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:25.135811 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:25.136001 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:25.136092 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:25.136098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:25.136964 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:25.136966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:25.137595 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:25.137605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:25.138035 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:25.138053 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:25.138117 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:25.139493 (+ 1376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:25.139506 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:25.141666 (+ 2160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":98,"threads_started":1}
W20260924 10:40:25.142052 11259 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:25.135305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:39005 (local address 127.2.8.62:41783)
0924 10:40:25.135450 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:25.135454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:25.135536 (+ 82us) server_negotiation.cc:407] Connection header received
0924 10:40:25.135649 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:25.135652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:25.135695 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:25.135769 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:25.136199 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:25.136866 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:25.137710 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:25.137936 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:25.138015 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:25.141945 (+ 3930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:39005: BlockingRecv error: recv got EOF from 127.2.8.2:39005 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 10:40:25 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:25 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:25.328751 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:25.322867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:48097)
0924 10:40:25.323171 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:25.323185 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:25.323280 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:25.323557 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:25.323559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:25.323736 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:25.323833 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:25.323838 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:25.324602 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:25.324605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:25.325308 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:25.325317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:25.325705 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:25.325719 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:25.325778 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:25.326706 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:25.326717 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:25.328634 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":109,"threads_started":1}
W20260924 10:40:25.328938 11259 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:25.322959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:48097 (local address 127.2.8.62:41783)
0924 10:40:25.323125 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:25.323130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:25.323253 (+ 123us) server_negotiation.cc:407] Connection header received
0924 10:40:25.323361 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:25.323363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:25.323408 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:25.323546 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:25.323962 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:25.324490 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:25.325402 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:25.325622 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:25.325701 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:25.328774 (+ 3073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:48097: BlockingRecv error: recv got EOF from 127.2.8.3:48097 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 10:40:25 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:25 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:25.986742 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:25.975777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:50889)
0924 10:40:25.976118 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:25.976135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:25.976300 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:25.976605 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:25.976608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:25.976814 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:25.976899 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:25.976905 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:25.977702 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:25.977709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:25.978266 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:25.978274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:25.978727 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:25.978745 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:25.978828 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:25.984688 (+ 5860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:25.984702 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:25.986620 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":102,"threads_started":1}
W20260924 10:40:25.986991 11263 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:25.975851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:50889 (local address 127.2.8.62:41783)
0924 10:40:25.976094 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:25.976099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:25.976235 (+ 136us) server_negotiation.cc:407] Connection header received
0924 10:40:25.976424 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:25.976429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:25.976482 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:25.976585 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:25.977041 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:25.977587 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:25.978388 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:25.978623 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:25.978742 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:25.986840 (+ 8098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:50889: BlockingRecv error: recv got EOF from 127.2.8.1:50889 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
Sep 24 10:40:26 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:26 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:26.148173 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:26.142523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:44293)
0924 10:40:26.142791 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:26.142804 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:26.142896 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:26.143098 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:26.143100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:26.143289 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:26.143379 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:26.143385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:26.144111 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:26.144114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:26.144752 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:26.144771 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:26.145338 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:26.145353 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:26.145413 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:26.145972 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:26.145985 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:26.148044 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":102,"threads_started":1}
W20260924 10:40:26.148348 11263 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:26.142571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:44293 (local address 127.2.8.62:41783)
0924 10:40:26.142737 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:26.142741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:26.142851 (+ 110us) server_negotiation.cc:407] Connection header received
0924 10:40:26.142966 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:26.142969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:26.143012 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:26.143081 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:26.143463 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:26.143992 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:26.144881 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:26.145217 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:26.145346 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:26.148212 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:44293: BlockingRecv error: recv got EOF from 127.2.8.2:44293 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 24 10:40:26 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:26 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 10:40:26.335062 11263 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:26.329480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:40953 (local address 127.2.8.62:41783)
0924 10:40:26.329626 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:26.329631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:26.329807 (+ 176us) server_negotiation.cc:407] Connection header received
0924 10:40:26.329955 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:26.329960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:26.330018 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:26.330103 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:26.330542 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:26.331091 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:26.331913 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:26.332147 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:26.332250 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:26.334946 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:40953: BlockingRecv error: recv got EOF from 127.2.8.3:40953 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:40:26.335074 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:26.329427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:40953)
0924 10:40:26.329707 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:26.329719 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:26.329862 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:26.330124 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:26.330127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:26.330335 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:26.330425 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:26.330430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:26.331197 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:26.331202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:26.331787 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:26.331794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:26.332237 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:26.332253 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:26.332339 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:26.332836 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:26.332847 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:26.334743 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":117,"threads_started":1}
Sep 24 10:40:26 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:26 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:26.993129 11266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:26.987731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:60473)
0924 10:40:26.988009 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:26.988023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:26.988105 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:26.988408 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:26.988411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:26.988589 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:26.988681 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:26.988687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:26.989562 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:26.989565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:26.990133 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:26.990141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:26.990587 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:26.990601 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:26.990657 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:26.991206 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:26.991218 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:26.992934 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":68,"threads_started":1}
W20260924 10:40:26.993333 11267 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:26.987770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:60473 (local address 127.2.8.62:41783)
0924 10:40:26.988062 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:26.988066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:26.988079 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:40:26.988228 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:26.988230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:26.988285 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:26.988382 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:26.988804 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:26.989459 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:26.990235 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:26.990503 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:26.990588 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:26.993154 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:60473: BlockingRecv error: recv got EOF from 127.2.8.1:60473 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":119,"threads_started":1}
Sep 24 10:40:27 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:27 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:27.154192 11268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:27.149091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:59261)
0924 10:40:27.149355 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:27.149368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:27.149461 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:27.149691 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:27.149693 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:27.149864 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:27.149962 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:27.149967 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:27.150649 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:27.150652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:27.151215 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:27.151222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:27.151605 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:27.151618 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:27.151679 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:27.152200 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:27.152212 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:27.154044 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":91,"threads_started":1}
W20260924 10:40:27.154284 11267 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:27.149105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:59261 (local address 127.2.8.62:41783)
0924 10:40:27.149251 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:27.149256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:27.149439 (+ 183us) server_negotiation.cc:407] Connection header received
0924 10:40:27.149546 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:27.149549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:27.149593 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:27.149660 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:27.150054 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:27.150563 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:27.151313 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:27.151531 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:27.151606 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:27.154166 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:59261: BlockingRecv error: recv got EOF from 127.2.8.2:59261 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 10:40:27 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:27 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:27.341528 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:27.335957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:52561)
0924 10:40:27.336254 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:27.336271 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:27.336408 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:27.336700 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:27.336702 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:27.336907 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:27.337044 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:27.337050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:27.337797 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:27.337803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:27.338374 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:27.338381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:27.338819 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:27.338837 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:27.338922 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:27.339448 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:27.339460 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:27.341356 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":97,"threads_started":1}
W20260924 10:40:27.341619 11267 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:27.336131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:52561 (local address 127.2.8.62:41783)
0924 10:40:27.336269 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:27.336274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:27.336350 (+ 76us) server_negotiation.cc:407] Connection header received
0924 10:40:27.336537 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:27.336540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:27.336592 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:27.336676 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:27.337157 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:27.337691 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:27.338507 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:27.338727 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:27.338831 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:27.341480 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:52561: BlockingRecv error: recv got EOF from 127.2.8.3:52561 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 10:40:27 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:27 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:27.999089 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:27.993861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:32851)
0924 10:40:27.994131 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:27.994144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:27.994236 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:27.994453 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:27.994455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:27.994627 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:27.994719 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:27.994724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:27.995435 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:27.995438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:27.995990 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:27.995998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:27.996409 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:27.996425 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:27.996493 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:27.997059 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:27.997083 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:27.998969 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
W20260924 10:40:27.999225 11271 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:27.993921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:32851 (local address 127.2.8.62:41783)
0924 10:40:27.994164 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:27.994167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:27.994194 (+ 27us) server_negotiation.cc:407] Connection header received
0924 10:40:27.994302 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:27.994304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:27.994344 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:27.994419 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:27.994825 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:27.995344 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:27.996085 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:27.996299 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:27.996372 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:27.999097 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:32851: BlockingRecv error: recv got EOF from 127.2.8.1:32851 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":57,"threads_started":1}
Sep 24 10:40:28 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:28 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:28.159972 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:28.154966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:59159)
0924 10:40:28.155210 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:28.155222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:28.155324 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:28.155538 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:28.155540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:28.155711 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:28.155799 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:28.155804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:28.156514 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:28.156516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:28.157133 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:28.157141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:28.157532 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:28.157545 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:28.157601 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:28.158072 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:28.158084 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:28.159850 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
W20260924 10:40:28.160099 11271 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:28.154999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:59159 (local address 127.2.8.62:41783)
0924 10:40:28.155120 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:28.155124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:28.155290 (+ 166us) server_negotiation.cc:407] Connection header received
0924 10:40:28.155401 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:28.155403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:28.155446 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:28.155510 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:28.155907 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:28.156427 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:28.157228 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:28.157440 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:28.157538 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:28.159978 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:59159: BlockingRecv error: recv got EOF from 127.2.8.2:59159 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 10:40:28 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:28 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:28.347797 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:28.342349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:52163)
0924 10:40:28.342638 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:28.342654 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:28.342767 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:28.343042 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:28.343051 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:28.343284 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:28.343388 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:28.343394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:28.344170 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:28.344172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:28.344798 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:28.344807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:28.345275 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:28.345290 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:28.345353 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:28.345910 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:28.345921 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:28.347685 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":107,"threads_started":1}
W20260924 10:40:28.347918 11271 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:28.342415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:52163 (local address 127.2.8.62:41783)
0924 10:40:28.342578 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:28.342586 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:28.342727 (+ 141us) server_negotiation.cc:407] Connection header received
0924 10:40:28.342866 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:28.342868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:28.342922 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:28.342983 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:28.343515 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:28.344068 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:28.344912 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:28.345181 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:28.345258 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:28.347805 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:52163: BlockingRecv error: recv got EOF from 127.2.8.3:52163 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 10:40:29 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:29 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:29.005529 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:28.999949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:42883)
0924 10:40:29.000219 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:29.000234 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:29.000321 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:29.000622 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:29.000624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:29.000801 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:29.000918 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:29.000923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:29.001742 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:29.001745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:29.002310 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:29.002319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:29.002766 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:29.002784 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:29.002847 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:29.003364 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:29.003378 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:29.005402 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":107,"threads_started":1}
W20260924 10:40:29.005604 11275 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:29.000041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:42883 (local address 127.2.8.62:41783)
0924 10:40:29.000297 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:29.000300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:29.000314 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:40:29.000441 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:29.000443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:29.000486 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:29.000568 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:29.001080 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:29.001619 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:29.002443 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:29.002681 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:29.002758 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:29.005499 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:42883: BlockingRecv error: recv got EOF from 127.2.8.1:42883 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":99,"threads_started":1}
Sep 24 10:40:29 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:29 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:29.166653 11276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:29.160891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:42565)
0924 10:40:29.161183 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:29.161198 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:29.161388 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:29.161687 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:29.161691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:29.161920 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:29.162019 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:29.162026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:29.162875 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:29.162882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:29.163512 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:29.163520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:29.164012 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:29.164031 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:29.164124 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:29.164648 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:29.164661 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:29.166475 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":114,"threads_started":1}
W20260924 10:40:29.166690 11275 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:29.160953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:42565 (local address 127.2.8.62:41783)
0924 10:40:29.161148 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:29.161153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:29.161280 (+ 127us) server_negotiation.cc:407] Connection header received
0924 10:40:29.161501 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:29.161507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:29.161561 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:29.161660 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:29.162146 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:29.162728 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:29.163634 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:29.163909 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:29.164026 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:29.166593 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:42565: BlockingRecv error: recv got EOF from 127.2.8.2:42565 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 24 10:40:29 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:29 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:29.353478 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:29.348563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:44349)
0924 10:40:29.348807 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:29.348821 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:29.348916 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:29.349215 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:29.349218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:29.349394 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:29.349493 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:29.349500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:29.350197 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:29.350199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:29.350747 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:29.350754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:29.351129 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:29.351142 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:29.351200 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:29.351668 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:29.351679 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:29.353368 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":101,"threads_started":1}
W20260924 10:40:29.353613 11275 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:29.348626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:44349 (local address 127.2.8.62:41783)
0924 10:40:29.348783 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:29.348788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:29.348889 (+ 101us) server_negotiation.cc:407] Connection header received
0924 10:40:29.349045 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:29.349047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:29.349092 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:29.349184 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:29.349592 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:29.350113 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:29.350839 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:29.351056 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:29.351127 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:29.353500 (+ 2373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:44349: BlockingRecv error: recv got EOF from 127.2.8.3:44349 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 10:40:30 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:30 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:30.012019 11278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:30.006393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:56353)
0924 10:40:30.006776 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:30.006790 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:30.006879 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:30.007151 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:30.007153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:30.007322 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:30.007415 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:30.007420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:30.008199 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:30.008202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:30.008760 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:30.008768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:30.009231 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:30.009247 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:30.009309 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:30.009838 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:30.009850 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:30.011877 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":193,"threads_started":1}
W20260924 10:40:30.012156 11279 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:30.006545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:56353 (local address 127.2.8.62:41783)
0924 10:40:30.006831 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:30.006835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:30.006851 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:40:30.006972 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:30.006975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:30.007022 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:30.007118 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:30.007530 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:30.008085 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:30.008884 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:30.009139 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:30.009219 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:30.012020 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:56353: BlockingRecv error: recv got EOF from 127.2.8.1:56353 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":96,"threads_started":1}
Sep 24 10:40:30 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:30 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:30.173542 11280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:30.167494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:55107)
0924 10:40:30.167856 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:30.167873 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:30.168018 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:30.168293 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:30.168296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:30.168538 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:30.168628 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:30.168634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:30.169789 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:30.169792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:30.170344 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:30.170352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:30.170803 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:30.170822 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:30.170907 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:30.171390 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:30.171401 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:30.173374 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":130,"threads_started":1}
W20260924 10:40:30.173667 11279 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:30.167667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:55107 (local address 127.2.8.62:41783)
0924 10:40:30.167848 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:30.167853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:30.167963 (+ 110us) server_negotiation.cc:407] Connection header received
0924 10:40:30.168117 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:30.168122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:30.168176 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:30.168267 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:30.168768 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:30.169670 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:30.170460 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:30.170708 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:30.170817 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:30.173537 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:55107: BlockingRecv error: recv got EOF from 127.2.8.2:55107 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 10:40:30 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:30 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:30.360297 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:30.354456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:47527)
0924 10:40:30.354795 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:30.354807 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:30.354944 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:30.355225 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:30.355228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:30.355432 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:30.355554 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:30.355560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:30.356318 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:30.356324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:30.356886 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:30.356894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:30.357371 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:30.357388 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:30.357482 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:30.357972 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:30.357984 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:30.360119 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":101,"threads_started":1}
W20260924 10:40:30.360419 11279 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:30.354564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:47527 (local address 127.2.8.62:41783)
0924 10:40:30.354733 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:30.354738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:30.354883 (+ 145us) server_negotiation.cc:407] Connection header received
0924 10:40:30.355057 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:30.355061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:30.355116 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:30.355204 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:30.355675 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:30.356210 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:30.357028 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:30.357275 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:30.357384 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:30.360297 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:47527: BlockingRecv error: recv got EOF from 127.2.8.3:47527 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 10:40:31 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:31 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:31.023746 11282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:31.012819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:45537)
0924 10:40:31.013135 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:31.013153 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:31.013293 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:31.013535 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:31.013537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:31.013713 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:31.013805 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:31.013810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:31.014522 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:31.014524 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:31.015067 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:31.015074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:31.015479 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:31.015496 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:31.015551 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:31.021350 (+ 5799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:31.021368 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:31.023591 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":69,"threads_started":1}
W20260924 10:40:31.023896 11283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:31.012950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:45537 (local address 127.2.8.62:41783)
0924 10:40:31.013260 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:31.013264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:31.013277 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:40:31.013379 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:31.013381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:31.013423 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:31.013490 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:31.013909 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:31.014432 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:31.015164 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:31.015383 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:31.015465 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:31.023739 (+ 8274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:45537: BlockingRecv error: recv got EOF from 127.2.8.1:45537 (error 108)
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":93,"threads_started":1}
Sep 24 10:40:31 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:31 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:31.179622 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:31.174398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:35409)
0924 10:40:31.174699 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:31.174712 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:31.174815 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:31.175049 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:31.175051 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:31.175220 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:31.175312 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:31.175317 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:31.176044 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:31.176047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:31.176596 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:31.176603 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:31.177045 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:31.177068 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:31.177134 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:31.177702 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:31.177714 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:31.179491 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":126,"threads_started":1}
W20260924 10:40:31.179781 11283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:31.174440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:35409 (local address 127.2.8.62:41783)
0924 10:40:31.174602 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:31.174606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:31.174790 (+ 184us) server_negotiation.cc:407] Connection header received
0924 10:40:31.174893 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:31.174895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:31.174940 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:31.175014 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:31.175418 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:31.175936 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:31.176699 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:31.176911 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:31.177061 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:31.179631 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:35409: BlockingRecv error: recv got EOF from 127.2.8.2:35409 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 10:40:31 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:31 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.3@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:31.366662 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:31.361207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.3:38411)
0924 10:40:31.361566 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:31.361586 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:31.361681 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:31.361914 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:31.361916 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:31.362078 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:31.362173 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:31.362178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:31.362923 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:31.362926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:31.363562 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:31.363571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:31.363970 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:31.363986 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:31.364052 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:31.364524 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:31.364535 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:31.366526 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"spinlock_wait_cycles":1280,"thread_start_us":183,"threads_started":1}
W20260924 10:40:31.366729 11283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:31.361256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:38411 (local address 127.2.8.62:41783)
0924 10:40:31.361378 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:31.361382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:31.361652 (+ 270us) server_negotiation.cc:407] Connection header received
0924 10:40:31.361758 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:31.361760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:31.361805 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:31.361873 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:31.362266 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:31.362799 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:31.363669 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:31.363876 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:31.363950 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:31.366641 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.3:38411: BlockingRecv error: recv got EOF from 127.2.8.3:38411 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 10:40:32 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:32 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.1@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:32.029973 11286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:32.024674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.1:56311)
0924 10:40:32.024937 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:32.024951 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:32.025066 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:32.025276 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:32.025278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:32.025447 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:32.025550 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:32.025555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.026292 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.026294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:32.026852 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:32.026860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.027274 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.027286 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:32.027345 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:32.027944 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:32.027965 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:32.029845 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":97,"threads_started":1}
W20260924 10:40:32.030121 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:32.024703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:56311 (local address 127.2.8.62:41783)
0924 10:40:32.024931 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:32.024934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:32.025046 (+ 112us) server_negotiation.cc:407] Connection header received
0924 10:40:32.025143 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:32.025145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:32.025189 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:32.025260 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:32.025662 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.026197 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.026966 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.027190 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.027269 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:32.029989 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.1:56311: BlockingRecv error: recv got EOF from 127.2.8.1:56311 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":83,"threads_started":1}
Sep 24 10:40:32 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 10:40:32 dist-test-slave-j38h krb5kdc[10229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.8.2@KRBTEST.COM for oryx/127.2.8.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 10:40:32.186157 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:32.180477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:41783 (local address 127.2.8.2:48261)
0924 10:40:32.180761 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:32.180782 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:32.180904 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:32.181197 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:32.181199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:32.181402 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:32.181499 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:32.181505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.182255 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.182261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:32.182826 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:32.182833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.183273 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.183290 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:32.183375 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:32.183867 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:32.183878 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:32.186046 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.8.62:41783: Server oryx/127.2.8.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
W20260924 10:40:32.186311 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:32.180546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:48261 (local address 127.2.8.62:41783)
0924 10:40:32.180707 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:32.180718 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:32.180859 (+ 141us) server_negotiation.cc:407] Connection header received
0924 10:40:32.181036 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:32.181039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:32.181096 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:32.181178 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:32.181618 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.182149 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.182944 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.183178 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.183284 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:32.186185 (+ 2901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.8.2:48261: BlockingRecv error: recv got EOF from 127.2.8.2:48261 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:40:32.264827 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 10812
I20260924 10:40:32.270176 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 10964
I20260924 10:40:32.275146 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 11096
I20260924 10:40:32.280186 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 10742
2026-09-24T10:40:32Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17987 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T10:40:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:40:32Z Disabled control of system clock
I20260924 10:40:32.309965 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:45267
--webserver_interface=127.2.8.62
--webserver_port=0
--builtin_ntp_servers=127.2.8.20:45511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:45267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 10:40:32.417484 11292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:32.417790 11292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:32.417857 11292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:32.421854 11292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:40:32.421940 11292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:32.421969 11292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:32.422014 11292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:40:32.422047 11292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:40:32.426724 11292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:45267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:45267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:32.427776 11292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:32.429173 11292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:32.437662 11297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:32.437628 11298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:32.437795 11300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:32.437839 11292 server_base.cc:1034] running on GCE node
I20260924 10:40:32.438441 11292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:32.439555 11292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:32.440734 11292 hybrid_clock.cc:648] HybridClock initialized: now 1790246432440704 us; error 44 us; skew 500 ppm
I20260924 10:40:32.442625 11292 webserver.cc:460] Webserver started at http://127.2.8.62:39325/ using document root <none> and password file <none>
I20260924 10:40:32.443198 11292 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:32.443291 11292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:32.443528 11292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:32.445567 11292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "e434a216324d49d4ba85bd6f7a767392"
format_stamp: "Formatted at 2026-09-24 10:40:32 on dist-test-slave-j38h"
server_key: "d70f2b13a422b65f93e80d5c2cc29dfa"
server_key_iv: "076ef385d39cd58efad783802da64ff7"
server_key_version: "encryptionkey@0"
I20260924 10:40:32.446110 11292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "e434a216324d49d4ba85bd6f7a767392"
format_stamp: "Formatted at 2026-09-24 10:40:32 on dist-test-slave-j38h"
server_key: "d70f2b13a422b65f93e80d5c2cc29dfa"
server_key_iv: "076ef385d39cd58efad783802da64ff7"
server_key_version: "encryptionkey@0"
I20260924 10:40:32.449591 11292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:40:32.451843 11306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:32.453027 11292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:40:32.453166 11292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "e434a216324d49d4ba85bd6f7a767392"
format_stamp: "Formatted at 2026-09-24 10:40:32 on dist-test-slave-j38h"
server_key: "d70f2b13a422b65f93e80d5c2cc29dfa"
server_key_iv: "076ef385d39cd58efad783802da64ff7"
server_key_version: "encryptionkey@0"
I20260924 10:40:32.453274 11292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:32.468961 11292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:32.469709 11292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:32.469901 11292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:32.476583 11292 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:45267
I20260924 10:40:32.476585 11358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:45267 every 8 connection(s)
I20260924 10:40:32.477600 11292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:40:32.480746 11359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:32.485093 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 11292
I20260924 10:40:32.485205 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:40:32.485533 2080 external_mini_cluster.cc:1428] Setting key fd2501398e089c75b9c2277606e8b7d0
I20260924 10:40:32.486471 11359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392: Bootstrap starting.
I20260924 10:40:32.488755 11359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:32.489540 11359 log.cc:824] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:32.491294 11359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392: No bootstrap required, opened a new log
I20260924 10:40:32.494561 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:32.486966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.2.8.62:45267)
0924 10:40:32.487580 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:32.487595 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:32.487630 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:40:32.488231 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:32.488247 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:32.488643 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:32.488972 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:32.489425 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.490426 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.491294 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.491819 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.492005 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:32.492838 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:32.492844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:32.492856 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:32.493312 (+ 456us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:32.493366 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:32.493387 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:32.493494 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:32.493618 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 10:40:32.493865 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":128,"threads_started":1}
I20260924 10:40:32.494839 11359 raft_consensus.cc:348] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e434a216324d49d4ba85bd6f7a767392" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45267 } }
I20260924 10:40:32.495076 11359 raft_consensus.cc:374] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:32.495162 11359 raft_consensus.cc:729] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e434a216324d49d4ba85bd6f7a767392, State: Initialized, Role: FOLLOWER
I20260924 10:40:32.495647 11359 consensus_queue.cc:260] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e434a216324d49d4ba85bd6f7a767392" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45267 } }
I20260924 10:40:32.495795 11359 raft_consensus.cc:388] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:40:32.495857 11359 raft_consensus.cc:482] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:40:32.495944 11359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:32.497066 11359 raft_consensus.cc:504] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e434a216324d49d4ba85bd6f7a767392" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45267 } }
I20260924 10:40:32.497344 11359 leader_election.cc:302] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e434a216324d49d4ba85bd6f7a767392; no voters:
I20260924 10:40:32.497573 11359 leader_election.cc:288] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:40:32.497712 11364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:32.497972 11364 raft_consensus.cc:686] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [term 1 LEADER]: Becoming Leader. State: Replica: e434a216324d49d4ba85bd6f7a767392, State: Running, Role: LEADER
I20260924 10:40:32.498322 11364 consensus_queue.cc:237] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e434a216324d49d4ba85bd6f7a767392" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45267 } }
I20260924 10:40:32.498771 11359 sys_catalog.cc:564] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:40:32.499958 11366 sys_catalog.cc:455] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e434a216324d49d4ba85bd6f7a767392. Latest consensus state: current_term: 1 leader_uuid: "e434a216324d49d4ba85bd6f7a767392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e434a216324d49d4ba85bd6f7a767392" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45267 } } }
I20260924 10:40:32.500080 11366 sys_catalog.cc:458] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:32.500530 11373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:40:32.501187 11365 sys_catalog.cc:455] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e434a216324d49d4ba85bd6f7a767392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e434a216324d49d4ba85bd6f7a767392" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45267 } } }
I20260924 10:40:32.501274 11365 sys_catalog.cc:458] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392 [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:32.503533 11373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:40:32.509390 11373 catalog_manager.cc:1347] Generated new cluster ID: a2a59b40ec734feb96b7291cf4b3c938
I20260924 10:40:32.509469 11373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:40:32.534485 11373 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:40:32.535516 11373 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:40:32.546923 11373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e434a216324d49d4ba85bd6f7a767392: Generated new TSK 0
I20260924 10:40:32.547729 11373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:40:32.562033 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:45267
--builtin_ntp_servers=127.2.8.20:45511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 10:40:32.674566 11383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:32.674839 11383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:32.674880 11383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:32.679121 11383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:32.679185 11383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:32.679240 11383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:40:32.683995 11383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:45267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:32.685196 11383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:32.686596 11383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:32.696296 11391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:32.696318 11388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:32.696302 11389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:32.696663 11383 server_base.cc:1034] running on GCE node
I20260924 10:40:32.697181 11383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:32.697825 11383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:32.699015 11383 hybrid_clock.cc:648] HybridClock initialized: now 1790246432698982 us; error 41 us; skew 500 ppm
I20260924 10:40:32.701177 11383 webserver.cc:460] Webserver started at http://127.2.8.1:45803/ using document root <none> and password file <none>
I20260924 10:40:32.701808 11383 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:32.701885 11383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:32.702203 11383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:32.705586 11383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "0512e033ff8f4589b2ad4a2939615774"
format_stamp: "Formatted at 2026-09-24 10:40:32 on dist-test-slave-j38h"
server_key: "c6dca9064c5cdc594bb7ace349d4fda1"
server_key_iv: "a68284cdb2a79a2997c04635c2fab9f1"
server_key_version: "encryptionkey@0"
I20260924 10:40:32.706184 11383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "0512e033ff8f4589b2ad4a2939615774"
format_stamp: "Formatted at 2026-09-24 10:40:32 on dist-test-slave-j38h"
server_key: "c6dca9064c5cdc594bb7ace349d4fda1"
server_key_iv: "a68284cdb2a79a2997c04635c2fab9f1"
server_key_version: "encryptionkey@0"
I20260924 10:40:32.710314 11383 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:40:32.712885 11397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:32.714183 11383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:40:32.714309 11383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "0512e033ff8f4589b2ad4a2939615774"
format_stamp: "Formatted at 2026-09-24 10:40:32 on dist-test-slave-j38h"
server_key: "c6dca9064c5cdc594bb7ace349d4fda1"
server_key_iv: "a68284cdb2a79a2997c04635c2fab9f1"
server_key_version: "encryptionkey@0"
I20260924 10:40:32.714427 11383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:32.734241 11383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:32.734974 11383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:32.735176 11383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:32.735796 11383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:32.736747 11383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:32.736817 11383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:32.736891 11383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:32.736932 11383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:32.745838 11383 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:45645
I20260924 10:40:32.745862 11510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:45645 every 8 connection(s)
I20260924 10:40:32.746829 11383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:40:32.748966 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 11383
I20260924 10:40:32.749145 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:40:32.749428 2080 external_mini_cluster.cc:1428] Setting key ecf6832c6676f673619d86c963fed78b
I20260924 10:40:32.751339 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:45267
--builtin_ntp_servers=127.2.8.20:45511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 10:40:32.757525 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:32.748570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:37337 (local address 127.2.8.62:45267)
0924 10:40:32.748743 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:32.748748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:32.750020 (+ 1272us) server_negotiation.cc:407] Connection header received
0924 10:40:32.751370 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:32.751374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:32.751431 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:32.751525 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:32.752894 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.753791 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.755038 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.755476 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.755609 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:32.756827 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:32.756832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:32.756839 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:32.756858 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:32.756886 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:32.756899 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:32.757009 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:32.757196 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 10:40:32.757298 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:40:32.758340 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:32.749105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45267 (local address 127.2.8.1:37337)
0924 10:40:32.749845 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:32.749897 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:32.751163 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:32.751846 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:32.751858 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:32.751976 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:32.752710 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:32.752736 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.753943 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.753946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:32.754872 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:32.754882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.755593 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.755641 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:32.755787 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:32.756492 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:40:32.756494 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:32.756536 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:40:32.756604 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:32.757044 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:32.757052 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:32.757056 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:40:32.757164 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 10:40:32.757490 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":128,"threads_started":1}
I20260924 10:40:32.759675 11511 heartbeater.cc:347] Connected to a master server at 127.2.8.62:45267
I20260924 10:40:32.759923 11511 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:32.760450 11511 heartbeater.cc:511] Master 127.2.8.62:45267 requested a full tablet report, sending...
I20260924 10:40:32.761981 11323 ts_manager.cc:194] Registered new tserver with Master: 0512e033ff8f4589b2ad4a2939615774 (127.2.8.1:45645)
I20260924 10:40:32.763098 11323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.8.1:37337
W20260924 10:40:32.870653 11514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:32.870914 11514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:32.870988 11514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:32.875008 11514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:32.875061 11514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:32.875159 11514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:40:32.880059 11514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:45267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:32.881258 11514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:32.882661 11514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:40:32.892387 11514 server_base.cc:1034] running on GCE node
W20260924 10:40:32.892334 11519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:32.892328 11520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:32.892696 11522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:32.893198 11514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:32.893841 11514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:32.895036 11514 hybrid_clock.cc:648] HybridClock initialized: now 1790246432895021 us; error 31 us; skew 500 ppm
I20260924 10:40:32.896838 11514 webserver.cc:460] Webserver started at http://127.2.8.2:45927/ using document root <none> and password file <none>
I20260924 10:40:32.897470 11514 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:32.897562 11514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:32.897789 11514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:32.900027 11514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "54ad5b3712fe435d9ff9897967f24365"
format_stamp: "Formatted at 2026-09-24 10:40:32 on dist-test-slave-j38h"
server_key: "c27a61e5f33dee3072dcceb7cc472ced"
server_key_iv: "1905632344ea43a1bd8fa980a0ad64a0"
server_key_version: "encryptionkey@0"
I20260924 10:40:32.900669 11514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "54ad5b3712fe435d9ff9897967f24365"
format_stamp: "Formatted at 2026-09-24 10:40:32 on dist-test-slave-j38h"
server_key: "c27a61e5f33dee3072dcceb7cc472ced"
server_key_iv: "1905632344ea43a1bd8fa980a0ad64a0"
server_key_version: "encryptionkey@0"
I20260924 10:40:32.904429 11514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:40:32.906955 11528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:32.908135 11514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:40:32.908267 11514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "54ad5b3712fe435d9ff9897967f24365"
format_stamp: "Formatted at 2026-09-24 10:40:32 on dist-test-slave-j38h"
server_key: "c27a61e5f33dee3072dcceb7cc472ced"
server_key_iv: "1905632344ea43a1bd8fa980a0ad64a0"
server_key_version: "encryptionkey@0"
I20260924 10:40:32.908380 11514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:32.925820 11514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:32.926493 11514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:32.926693 11514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:32.927268 11514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:32.928203 11514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:32.928273 11514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:32.928346 11514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:32.928390 11514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:32.937208 11514 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:38493
I20260924 10:40:32.937287 11641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:38493 every 8 connection(s)
I20260924 10:40:32.938138 11514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:40:32.947130 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:32.939984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:46653 (local address 127.2.8.62:45267)
0924 10:40:32.940158 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:32.940166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:32.941268 (+ 1102us) server_negotiation.cc:407] Connection header received
0924 10:40:32.942632 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:32.942635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:32.942684 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:32.942763 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:32.943663 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.944186 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.945375 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.945607 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.945691 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:32.946733 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:32.946736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:32.946737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:32.946748 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:32.946767 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:32.946774 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:32.946833 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:32.946969 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 10:40:32.947016 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 10:40:32.947963 11644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:32.940400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45267 (local address 127.2.8.2:46653)
0924 10:40:32.941116 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:32.941156 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:32.942384 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:32.942928 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:32.942934 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:32.943017 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:32.943498 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:32.943534 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.944294 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.944296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:32.945265 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:32.945274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:32.945698 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:32.945743 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:32.945876 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:32.946492 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:40:32.946494 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:32.946537 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:40:32.946597 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:32.946847 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:32.946852 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:32.946856 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:40:32.946950 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 10:40:32.947161 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":118,"threads_started":1}
I20260924 10:40:32.948112 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 11514
I20260924 10:40:32.948261 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:40:32.948593 2080 external_mini_cluster.cc:1428] Setting key e8504bcfd917c41a58f6e49de66d06c7
I20260924 10:40:32.949204 11642 heartbeater.cc:347] Connected to a master server at 127.2.8.62:45267
I20260924 10:40:32.949445 11642 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:32.950032 11642 heartbeater.cc:511] Master 127.2.8.62:45267 requested a full tablet report, sending...
I20260924 10:40:32.950635 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:45267
--builtin_ntp_servers=127.2.8.20:45511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 10:40:32.951097 11323 ts_manager.cc:194] Registered new tserver with Master: 54ad5b3712fe435d9ff9897967f24365 (127.2.8.2:38493)
I20260924 10:40:32.951653 11323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.8.2:46653
W20260924 10:40:33.062134 11645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:33.062402 11645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:33.062438 11645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:33.066582 11645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:33.066638 11645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:33.066691 11645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:40:33.071434 11645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:45267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:33.072515 11645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:33.073890 11645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:33.083458 11650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:33.083535 11651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:33.083585 11645 server_base.cc:1034] running on GCE node
W20260924 10:40:33.083459 11653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:33.084057 11645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:33.084652 11645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:33.085848 11645 hybrid_clock.cc:648] HybridClock initialized: now 1790246433085797 us; error 64 us; skew 500 ppm
I20260924 10:40:33.087677 11645 webserver.cc:460] Webserver started at http://127.2.8.3:35373/ using document root <none> and password file <none>
I20260924 10:40:33.088274 11645 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:33.088366 11645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:33.088598 11645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:33.090715 11645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "bc8eb73fb26b4069815caedb2a839433"
format_stamp: "Formatted at 2026-09-24 10:40:33 on dist-test-slave-j38h"
server_key: "2b6db8a5ccba7c975d0186b558d8286b"
server_key_iv: "e4edea7f7c0ef19e5c2d41d2f0f89b2c"
server_key_version: "encryptionkey@0"
I20260924 10:40:33.091274 11645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "bc8eb73fb26b4069815caedb2a839433"
format_stamp: "Formatted at 2026-09-24 10:40:33 on dist-test-slave-j38h"
server_key: "2b6db8a5ccba7c975d0186b558d8286b"
server_key_iv: "e4edea7f7c0ef19e5c2d41d2f0f89b2c"
server_key_version: "encryptionkey@0"
I20260924 10:40:33.094933 11645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 10:40:33.097409 11659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:33.098593 11645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:40:33.098724 11645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "bc8eb73fb26b4069815caedb2a839433"
format_stamp: "Formatted at 2026-09-24 10:40:33 on dist-test-slave-j38h"
server_key: "2b6db8a5ccba7c975d0186b558d8286b"
server_key_iv: "e4edea7f7c0ef19e5c2d41d2f0f89b2c"
server_key_version: "encryptionkey@0"
I20260924 10:40:33.098835 11645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:33.138010 11645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:33.138746 11645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:33.138952 11645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:33.139549 11645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:33.140496 11645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:33.140575 11645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:33.140647 11645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:33.140697 11645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:33.149420 11645 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:38973
I20260924 10:40:33.149565 11772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:38973 every 8 connection(s)
I20260924 10:40:33.150338 11645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:40:33.157347 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 11645
I20260924 10:40:33.157464 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:40:33.157779 2080 external_mini_cluster.cc:1428] Setting key 0147928fe69056bd772bac9f72f20241
I20260924 10:40:33.160257 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:33.152144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:35883 (local address 127.2.8.62:45267)
0924 10:40:33.152296 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:33.152300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:33.153479 (+ 1179us) server_negotiation.cc:407] Connection header received
0924 10:40:33.154870 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:33.154873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:33.154924 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:33.155012 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:33.155918 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:33.156614 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:33.158027 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:33.158315 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:33.158416 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:33.159762 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:33.159766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:33.159772 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:33.159788 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:33.159812 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:33.159823 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:33.159891 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:33.160101 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 10:40:33.160150 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:40:33.161206 11775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:33.152636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45267 (local address 127.2.8.3:35883)
0924 10:40:33.153327 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:33.153368 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:33.154634 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:33.155182 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:33.155188 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:33.155274 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:33.155783 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:33.155796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:33.156732 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:33.156735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:33.157900 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:33.157913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:33.158414 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:33.158484 (+ 70us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:33.158725 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:33.159436 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:40:33.159438 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:33.159492 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:40:33.159582 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:33.159913 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:33.159922 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:33.159930 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:40:33.160060 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 10:40:33.160292 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":104,"threads_started":1}
I20260924 10:40:33.162288 11773 heartbeater.cc:347] Connected to a master server at 127.2.8.62:45267
I20260924 10:40:33.162549 11773 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:33.163060 11773 heartbeater.cc:511] Master 127.2.8.62:45267 requested a full tablet report, sending...
I20260924 10:40:33.164052 11323 ts_manager.cc:194] Registered new tserver with Master: bc8eb73fb26b4069815caedb2a839433 (127.2.8.3:38973)
I20260924 10:40:33.164669 11323 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.8.3:35883
I20260924 10:40:33.171515 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 10:40:33.174190 11362 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:33.173379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48602 (local address 127.2.8.62:45267)
0924 10:40:33.173516 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:33.173520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:33.173661 (+ 141us) server_negotiation.cc:407] Connection header received
0924 10:40:33.173747 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:33.173749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:33.173786 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:33.173851 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:33.174049 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48602: BlockingRecv error: recv got EOF from 127.0.0.1:48602 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:40:33.174806 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 11383
I20260924 10:40:33.180461 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 11514
I20260924 10:40:33.185751 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 11645
I20260924 10:40:33.190672 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 11292
2026-09-24T10:40:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (909 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T10:40:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:40:33Z Disabled control of system clock
I20260924 10:40:33.219277 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:44421
--webserver_interface=127.2.8.62
--webserver_port=0
--builtin_ntp_servers=127.2.8.20:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:44421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 10:40:33.325933 11784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:33.326210 11784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:33.326248 11784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:33.330130 11784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:40:33.330188 11784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:33.330204 11784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:33.330219 11784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:40:33.330235 11784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:40:33.334972 11784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:44421
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:44421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:33.335978 11784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:33.337278 11784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:33.345319 11790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:33.345357 11789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:33.345319 11792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:33.345662 11784 server_base.cc:1034] running on GCE node
I20260924 10:40:33.346174 11784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:33.347268 11784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:33.348450 11784 hybrid_clock.cc:648] HybridClock initialized: now 1790246433348420 us; error 43 us; skew 500 ppm
I20260924 10:40:33.350244 11784 webserver.cc:460] Webserver started at http://127.2.8.62:44711/ using document root <none> and password file <none>
I20260924 10:40:33.350803 11784 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:33.350891 11784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:33.351118 11784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:33.353128 11784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "8fde9480d69549c8acb7cf671c6644bb"
format_stamp: "Formatted at 2026-09-24 10:40:33 on dist-test-slave-j38h"
server_key: "d6f727a7a573e0dc40af82c3e07f8384"
server_key_iv: "b4a2adc777f06ae1ecfef9d0e050536d"
server_key_version: "encryptionkey@0"
I20260924 10:40:33.353682 11784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "8fde9480d69549c8acb7cf671c6644bb"
format_stamp: "Formatted at 2026-09-24 10:40:33 on dist-test-slave-j38h"
server_key: "d6f727a7a573e0dc40af82c3e07f8384"
server_key_iv: "b4a2adc777f06ae1ecfef9d0e050536d"
server_key_version: "encryptionkey@0"
I20260924 10:40:33.357213 11784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 10:40:33.359493 11798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:33.360654 11784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260924 10:40:33.360787 11784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "8fde9480d69549c8acb7cf671c6644bb"
format_stamp: "Formatted at 2026-09-24 10:40:33 on dist-test-slave-j38h"
server_key: "d6f727a7a573e0dc40af82c3e07f8384"
server_key_iv: "b4a2adc777f06ae1ecfef9d0e050536d"
server_key_version: "encryptionkey@0"
I20260924 10:40:33.360894 11784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:33.365356 11784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:33.365955 11784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:33.366125 11784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:33.372545 11784 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:44421
I20260924 10:40:33.372551 11850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:44421 every 8 connection(s)
I20260924 10:40:33.373530 11784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:40:33.374490 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 11784
I20260924 10:40:33.374624 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:40:33.374938 2080 external_mini_cluster.cc:1428] Setting key fcdd0d8d8f59caf66a85a8e9ca55a9ae
I20260924 10:40:33.376751 11851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:33.380795 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:33.376276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.2.8.62:44421)
0924 10:40:33.377359 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:33.377373 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:33.377407 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:40:33.378031 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:33.378046 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:33.378372 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:33.378650 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:33.379568 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:33.379573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:33.379584 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:33.379593 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:33.379643 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:33.379659 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:33.379765 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:33.379870 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 10:40:33.380144 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":601,"threads_started":1}
I20260924 10:40:33.383206 11851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb: Bootstrap starting.
I20260924 10:40:33.385329 11851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:33.386140 11851 log.cc:824] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:33.387867 11851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb: No bootstrap required, opened a new log
I20260924 10:40:33.391212 11851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fde9480d69549c8acb7cf671c6644bb" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44421 } }
I20260924 10:40:33.391474 11851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:33.391547 11851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fde9480d69549c8acb7cf671c6644bb, State: Initialized, Role: FOLLOWER
I20260924 10:40:33.391984 11851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fde9480d69549c8acb7cf671c6644bb" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44421 } }
I20260924 10:40:33.392129 11851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:40:33.392194 11851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:40:33.392290 11851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:33.393229 11851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fde9480d69549c8acb7cf671c6644bb" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44421 } }
I20260924 10:40:33.393556 11851 leader_election.cc:302] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fde9480d69549c8acb7cf671c6644bb; no voters:
I20260924 10:40:33.393847 11851 leader_election.cc:288] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:40:33.393999 11856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:33.394254 11856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [term 1 LEADER]: Becoming Leader. State: Replica: 8fde9480d69549c8acb7cf671c6644bb, State: Running, Role: LEADER
I20260924 10:40:33.394546 11856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fde9480d69549c8acb7cf671c6644bb" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44421 } }
I20260924 10:40:33.395061 11851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:40:33.396164 11857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fde9480d69549c8acb7cf671c6644bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fde9480d69549c8acb7cf671c6644bb" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44421 } } }
I20260924 10:40:33.396292 11857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:33.396562 11858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fde9480d69549c8acb7cf671c6644bb. Latest consensus state: current_term: 1 leader_uuid: "8fde9480d69549c8acb7cf671c6644bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fde9480d69549c8acb7cf671c6644bb" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44421 } } }
I20260924 10:40:33.396646 11858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:33.396914 11865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:40:33.399456 11865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:40:33.404179 11865 catalog_manager.cc:1347] Generated new cluster ID: 42c302cbf11043ea8debc7f5a43bf4f7
I20260924 10:40:33.404240 11865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:40:33.432464 11865 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:40:33.432797 11865 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:40:33.439782 11865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fde9480d69549c8acb7cf671c6644bb: Generated new TSK 0
I20260924 10:40:33.440567 11865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:40:33.444120 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:44421
--builtin_ntp_servers=127.2.8.20:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 10:40:33.554430 11875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:33.554706 11875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:33.554775 11875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:33.558657 11875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:33.558737 11875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:33.558826 11875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:40:33.563433 11875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:33.564497 11875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:33.565866 11875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:33.575002 11880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:33.575081 11883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:33.575003 11881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:33.575224 11875 server_base.cc:1034] running on GCE node
I20260924 10:40:33.575668 11875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:33.576284 11875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:33.577462 11875 hybrid_clock.cc:648] HybridClock initialized: now 1790246433577426 us; error 48 us; skew 500 ppm
I20260924 10:40:33.579259 11875 webserver.cc:460] Webserver started at http://127.2.8.1:32947/ using document root <none> and password file <none>
I20260924 10:40:33.579833 11875 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:33.579921 11875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:33.580161 11875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:33.582175 11875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "63182e70db8140edadfd3a9941870c51"
format_stamp: "Formatted at 2026-09-24 10:40:33 on dist-test-slave-j38h"
server_key: "ac3bad370ff331afa0fbc1e550896a94"
server_key_iv: "a7a3dcb78b1b5c03a56a65c2c1abd8b6"
server_key_version: "encryptionkey@0"
I20260924 10:40:33.582719 11875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "63182e70db8140edadfd3a9941870c51"
format_stamp: "Formatted at 2026-09-24 10:40:33 on dist-test-slave-j38h"
server_key: "ac3bad370ff331afa0fbc1e550896a94"
server_key_iv: "a7a3dcb78b1b5c03a56a65c2c1abd8b6"
server_key_version: "encryptionkey@0"
I20260924 10:40:33.586164 11875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:40:33.588469 11889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:33.589743 11875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:40:33.589875 11875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "63182e70db8140edadfd3a9941870c51"
format_stamp: "Formatted at 2026-09-24 10:40:33 on dist-test-slave-j38h"
server_key: "ac3bad370ff331afa0fbc1e550896a94"
server_key_iv: "a7a3dcb78b1b5c03a56a65c2c1abd8b6"
server_key_version: "encryptionkey@0"
I20260924 10:40:33.589990 11875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:33.594619 11875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:33.595204 11875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:33.595402 11875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:33.595999 11875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:33.596935 11875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:33.597023 11875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:33.597091 11875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:33.597123 11875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:33.605687 11875 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:36263
I20260924 10:40:33.605731 12002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:36263 every 8 connection(s)
I20260924 10:40:33.606562 11875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:40:33.610061 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 11875
I20260924 10:40:33.610175 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:40:33.610530 2080 external_mini_cluster.cc:1428] Setting key 8611871d25d91b858ad1ebcf7aa340be
I20260924 10:40:33.612614 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:44421
--builtin_ntp_servers=127.2.8.20:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 10:40:33.612911 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:33.608254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:53969 (local address 127.2.8.62:44421)
0924 10:40:33.608394 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:33.608397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:33.609458 (+ 1061us) server_negotiation.cc:407] Connection header received
0924 10:40:33.611208 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:33.611211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:33.611254 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:33.611334 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:33.612443 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:33.612449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:33.612452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:33.612464 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:33.612485 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:33.612493 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:33.612597 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:33.612749 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 10:40:33.612789 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:40:33.613771 12005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:33.608647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:44421 (local address 127.2.8.1:53969)
0924 10:40:33.609320 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:33.609358 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:33.610970 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:33.611499 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:33.611506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:33.611594 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:33.612198 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:40:33.612200 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:33.612238 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:40:33.612302 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:33.612584 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:33.612592 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:33.612597 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:40:33.612792 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 10:40:33.613042 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":104,"threads_started":1}
I20260924 10:40:33.614923 12003 heartbeater.cc:347] Connected to a master server at 127.2.8.62:44421
I20260924 10:40:33.615134 12003 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:33.615511 12003 heartbeater.cc:511] Master 127.2.8.62:44421 requested a full tablet report, sending...
I20260924 10:40:33.617072 11815 ts_manager.cc:194] Registered new tserver with Master: 63182e70db8140edadfd3a9941870c51 (127.2.8.1:36263)
W20260924 10:40:33.724082 12006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:33.724344 12006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:33.724383 12006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:33.728459 12006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:33.728529 12006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:33.728585 12006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:40:33.733426 12006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:33.734553 12006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:33.735967 12006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:33.745543 12012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:33.745602 12006 server_base.cc:1034] running on GCE node
W20260924 10:40:33.745527 12011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:33.745546 12014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:33.746106 12006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:33.746706 12006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:33.747862 12006 hybrid_clock.cc:648] HybridClock initialized: now 1790246433747848 us; error 25 us; skew 500 ppm
I20260924 10:40:33.749632 12006 webserver.cc:460] Webserver started at http://127.2.8.2:39685/ using document root <none> and password file <none>
I20260924 10:40:33.750193 12006 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:33.750252 12006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:33.750514 12006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:33.752580 12006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "f35a32013971438dbfe89629beb50b98"
format_stamp: "Formatted at 2026-09-24 10:40:33 on dist-test-slave-j38h"
server_key: "88e1a04c675d30f78e89d0ddec4c55a2"
server_key_iv: "01493dfc8ac7e91f9afa732a0fe34caa"
server_key_version: "encryptionkey@0"
I20260924 10:40:33.753167 12006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "f35a32013971438dbfe89629beb50b98"
format_stamp: "Formatted at 2026-09-24 10:40:33 on dist-test-slave-j38h"
server_key: "88e1a04c675d30f78e89d0ddec4c55a2"
server_key_iv: "01493dfc8ac7e91f9afa732a0fe34caa"
server_key_version: "encryptionkey@0"
I20260924 10:40:33.756691 12006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:40:33.759188 12020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:33.760424 12006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:40:33.760550 12006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "f35a32013971438dbfe89629beb50b98"
format_stamp: "Formatted at 2026-09-24 10:40:33 on dist-test-slave-j38h"
server_key: "88e1a04c675d30f78e89d0ddec4c55a2"
server_key_iv: "01493dfc8ac7e91f9afa732a0fe34caa"
server_key_version: "encryptionkey@0"
I20260924 10:40:33.760653 12006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:33.765285 12006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:33.765897 12006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:33.766083 12006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:33.766625 12006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:33.767531 12006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:33.767577 12006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:33.767637 12006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:33.767673 12006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:33.776582 12006 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:33431
I20260924 10:40:33.776604 12133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:33431 every 8 connection(s)
I20260924 10:40:33.777516 12006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:40:33.778807 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 12006
I20260924 10:40:33.778944 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:40:33.779289 2080 external_mini_cluster.cc:1428] Setting key a2cb8a664d771adda4a3faf7c6667f88
I20260924 10:40:33.781512 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:44421
--builtin_ntp_servers=127.2.8.20:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 10:40:33.785017 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:33.779548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:33537 (local address 127.2.8.62:44421)
0924 10:40:33.779700 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:33.779707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:33.781159 (+ 1452us) server_negotiation.cc:407] Connection header received
0924 10:40:33.783083 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:33.783090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:33.783138 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:33.783233 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:33.784460 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:33.784467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:33.784473 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:33.784489 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:33.784516 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:33.784529 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:33.784629 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:33.784751 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 10:40:33.784800 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:40:33.786104 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:33.780107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:44421 (local address 127.2.8.2:33537)
0924 10:40:33.780980 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:33.781053 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:33.782831 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:33.783394 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:33.783447 (+ 53us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:33.783541 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:33.784188 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:40:33.784190 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:33.784231 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:40:33.784302 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:33.784618 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:33.784624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:33.784628 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 10:40:33.784759 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 10:40:33.784950 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":143,"threads_started":1}
I20260924 10:40:33.787298 12134 heartbeater.cc:347] Connected to a master server at 127.2.8.62:44421
I20260924 10:40:33.787530 12134 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:33.787942 12134 heartbeater.cc:511] Master 127.2.8.62:44421 requested a full tablet report, sending...
I20260924 10:40:33.788965 11815 ts_manager.cc:194] Registered new tserver with Master: f35a32013971438dbfe89629beb50b98 (127.2.8.2:33431)
W20260924 10:40:33.891727 12137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:33.891979 12137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:33.892017 12137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:33.896518 12137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:33.896596 12137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:33.896653 12137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:40:33.901527 12137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:33.902552 12137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:33.903905 12137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:33.913303 12143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:33.913306 12145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:33.913306 12142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:33.913540 12137 server_base.cc:1034] running on GCE node
I20260924 10:40:33.914000 12137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:33.914906 12137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:33.916173 12137 hybrid_clock.cc:648] HybridClock initialized: now 1790246433916159 us; error 32 us; skew 500 ppm
I20260924 10:40:33.918169 12137 webserver.cc:460] Webserver started at http://127.2.8.3:36813/ using document root <none> and password file <none>
I20260924 10:40:33.918751 12137 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:33.918843 12137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:33.919072 12137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:33.921187 12137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "f6b5a1f740d247d083a61543306ffd38"
format_stamp: "Formatted at 2026-09-24 10:40:33 on dist-test-slave-j38h"
server_key: "1905353056ed7cd904a1bc3bb99bd279"
server_key_iv: "edd80b385273ffe7a30d258052570058"
server_key_version: "encryptionkey@0"
I20260924 10:40:33.921877 12137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "f6b5a1f740d247d083a61543306ffd38"
format_stamp: "Formatted at 2026-09-24 10:40:33 on dist-test-slave-j38h"
server_key: "1905353056ed7cd904a1bc3bb99bd279"
server_key_iv: "edd80b385273ffe7a30d258052570058"
server_key_version: "encryptionkey@0"
I20260924 10:40:33.925640 12137 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:40:33.928229 12151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:33.929540 12137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 10:40:33.929666 12137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "f6b5a1f740d247d083a61543306ffd38"
format_stamp: "Formatted at 2026-09-24 10:40:33 on dist-test-slave-j38h"
server_key: "1905353056ed7cd904a1bc3bb99bd279"
server_key_iv: "edd80b385273ffe7a30d258052570058"
server_key_version: "encryptionkey@0"
I20260924 10:40:33.929778 12137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:33.934245 12137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:33.934825 12137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:33.935002 12137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:33.935523 12137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:33.936384 12137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:33.936457 12137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:33.936532 12137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:33.936581 12137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:33.945816 12137 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:36655
I20260924 10:40:33.945835 12264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:36655 every 8 connection(s)
I20260924 10:40:33.946681 12137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:40:33.948418 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 12137
I20260924 10:40:33.948523 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:40:33.948814 2080 external_mini_cluster.cc:1428] Setting key 332f1f1a7cc756f32e8b961193b1f853
I20260924 10:40:33.952665 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:33.948481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:41775 (local address 127.2.8.62:44421)
0924 10:40:33.949152 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:33.949157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:33.949908 (+ 751us) server_negotiation.cc:407] Connection header received
0924 10:40:33.951184 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:33.951187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:33.951221 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:33.951284 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:33.952336 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:33.952338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:33.952340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:40:33.952349 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:33.952365 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:40:33.952371 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:33.952435 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:33.952532 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 10:40:33.952566 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556}
I20260924 10:40:33.953568 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:33.949026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:44421 (local address 127.2.8.3:41775)
0924 10:40:33.949751 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:33.949801 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:33.951007 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:33.951411 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:33.951418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:33.951502 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:33.952108 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:40:33.952110 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:33.952148 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:40:33.952214 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:33.952426 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:33.952430 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:33.952433 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 10:40:33.952546 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 10:40:33.952767 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":134,"threads_started":1}
I20260924 10:40:33.954675 12265 heartbeater.cc:347] Connected to a master server at 127.2.8.62:44421
I20260924 10:40:33.954906 12265 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:33.955286 12265 heartbeater.cc:511] Master 127.2.8.62:44421 requested a full tablet report, sending...
I20260924 10:40:33.956240 11815 ts_manager.cc:194] Registered new tserver with Master: f6b5a1f740d247d083a61543306ffd38 (127.2.8.3:36655)
I20260924 10:40:33.962692 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 10:40:33.965989 11854 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:40:33.964844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.2.8.62:44421)
0924 10:40:33.964942 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:33.964947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:33.965431 (+ 484us) server_negotiation.cc:407] Connection header received
0924 10:40:33.965544 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:33.965547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:33.965579 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:33.965643 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:33.965857 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47522: BlockingRecv error: recv got EOF from 127.0.0.1:47522 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 10:40:33.966198 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 11875
I20260924 10:40:33.972280 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 12006
I20260924 10:40:33.977434 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 12137
I20260924 10:40:33.982559 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 11784
2026-09-24T10:40:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (791 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:40:34 dist-test-slave-j38h krb5kdc[12275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:40:34 dist-test-slave-j38h krb5kdc[12275](info): set up 2 sockets
Sep 24 10:40:34 dist-test-slave-j38h krb5kdc[12275](info): commencing operation
krb5kdc: starting...
W20260924 10:40:35.038641 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:40:35 dist-test-slave-j38h krb5kdc[12275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:40:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:40:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:40:35.162050 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:43745
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:43745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:35.271948 12288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:35.272238 12288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:35.272325 12288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:35.276336 12288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:40:35.276419 12288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:35.276456 12288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:35.276520 12288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:40:35.276575 12288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:40:35.281788 12288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:43745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:43745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:35.282847 12288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:35.283926 12288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:35.292073 12294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:35.292081 12293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:35.292157 12296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:35.292264 12288 server_base.cc:1034] running on GCE node
I20260924 10:40:35.292865 12288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:35.293989 12288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:35.295171 12288 hybrid_clock.cc:648] HybridClock initialized: now 1790246435295158 us; error 25 us; skew 500 ppm
Sep 24 10:40:35 dist-test-slave-j38h krb5kdc[12275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:35.297892 12288 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:40:35.298696 12288 webserver.cc:460] Webserver started at http://127.2.8.62:45953/ using document root <none> and password file <none>
I20260924 10:40:35.299234 12288 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:35.299324 12288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:35.299546 12288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:35.301563 12288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "1087361dc2bb4b75b51c86331189b8ea"
format_stamp: "Formatted at 2026-09-24 10:40:35 on dist-test-slave-j38h"
server_key: "a5643d54c0e7e69319231c874484689c"
server_key_iv: "e002c4bdb1e281cdbd79895667eee694"
server_key_version: "encryptionkey@0"
I20260924 10:40:35.302096 12288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "1087361dc2bb4b75b51c86331189b8ea"
format_stamp: "Formatted at 2026-09-24 10:40:35 on dist-test-slave-j38h"
server_key: "a5643d54c0e7e69319231c874484689c"
server_key_iv: "e002c4bdb1e281cdbd79895667eee694"
server_key_version: "encryptionkey@0"
I20260924 10:40:35.305547 12288 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:40:35.307920 12303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:35.309181 12288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:40:35.309314 12288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "1087361dc2bb4b75b51c86331189b8ea"
format_stamp: "Formatted at 2026-09-24 10:40:35 on dist-test-slave-j38h"
server_key: "a5643d54c0e7e69319231c874484689c"
server_key_iv: "e002c4bdb1e281cdbd79895667eee694"
server_key_version: "encryptionkey@0"
I20260924 10:40:35.309427 12288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:35.338308 12288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:35.341679 12288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:35.341884 12288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:35.348459 12288 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:43745
I20260924 10:40:35.348490 12355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:43745 every 8 connection(s)
I20260924 10:40:35.349437 12288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:40:35.352110 12356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:35.357434 12356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea: Bootstrap starting.
I20260924 10:40:35.358335 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 12288
I20260924 10:40:35.358464 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:40:35.358743 2080 external_mini_cluster.cc:1428] Setting key 8f4e177eeacdccb9330936ad6eae42b6
I20260924 10:40:35.360133 12356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:35.360838 12356 log.cc:824] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:35.362666 12356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea: No bootstrap required, opened a new log
I20260924 10:40:35.365489 12356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1087361dc2bb4b75b51c86331189b8ea" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43745 } }
I20260924 10:40:35.365720 12356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:35.365800 12356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1087361dc2bb4b75b51c86331189b8ea, State: Initialized, Role: FOLLOWER
I20260924 10:40:35.366312 12356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1087361dc2bb4b75b51c86331189b8ea" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43745 } }
I20260924 10:40:35.366492 12356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:40:35.366565 12356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:40:35.366680 12356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [term 0 FOLLOWER]: Advancing to term 1
Sep 24 10:40:35 dist-test-slave-j38h krb5kdc[12275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:40:35.367671 12356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1087361dc2bb4b75b51c86331189b8ea" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43745 } }
I20260924 10:40:35.368024 12356 leader_election.cc:302] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1087361dc2bb4b75b51c86331189b8ea; no voters:
I20260924 10:40:35.368341 12356 leader_election.cc:288] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:40:35.368439 12361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:35.368714 12361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [term 1 LEADER]: Becoming Leader. State: Replica: 1087361dc2bb4b75b51c86331189b8ea, State: Running, Role: LEADER
I20260924 10:40:35.369072 12361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1087361dc2bb4b75b51c86331189b8ea" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43745 } }
I20260924 10:40:35.369923 12356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:40:35.370630 12362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1087361dc2bb4b75b51c86331189b8ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1087361dc2bb4b75b51c86331189b8ea" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43745 } } }
I20260924 10:40:35.370739 12362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:35.371109 12363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1087361dc2bb4b75b51c86331189b8ea. Latest consensus state: current_term: 1 leader_uuid: "1087361dc2bb4b75b51c86331189b8ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1087361dc2bb4b75b51c86331189b8ea" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43745 } } }
I20260924 10:40:35.371239 12363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea [sys.catalog]: This master's current role is: LEADER
I20260924 10:40:35.372043 12369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:40:35.374056 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:35.359824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.2.8.62:43745)
0924 10:40:35.360221 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:35.360256 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:35.360304 (+ 48us) server_negotiation.cc:407] Connection header received
0924 10:40:35.360905 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:35.360921 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:35.361320 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:35.361652 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:35.362440 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:35.363455 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:35.364287 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:35.365253 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:35.365429 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:35.368204 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:35.368210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:35.368227 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:35.368260 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:35.370397 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:35.370961 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:35.370967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:35.370973 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:35.371114 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:35.371428 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:35.371433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:35.371434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:35.372007 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:35.372223 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:35.372787 (+ 564us) server_negotiation.cc:299] Negotiation successful
0924 10:40:35.373125 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":128,"threads_started":1}
I20260924 10:40:35.375394 12369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:40:35.380914 12369 catalog_manager.cc:1347] Generated new cluster ID: bfa33be4b8dc448db7fbe9b820f3ed7f
I20260924 10:40:35.381006 12369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:40:35.393913 12369 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:40:35.395088 12369 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:40:35.406602 12369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea: Generated new TSK 0
I20260924 10:40:35.407401 12369 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:40:35.437372 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43745
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:35.548638 12382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:35.548924 12382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:35.549068 12382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:35.553077 12382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:35.553166 12382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:35.553257 12382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:40:35.558439 12382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:35.559456 12382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:35.560468 12382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:35.569777 12390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:35.569796 12387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:35.569782 12388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:35.570139 12382 server_base.cc:1034] running on GCE node
I20260924 10:40:35.570520 12382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:35.571228 12382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:35.572394 12382 hybrid_clock.cc:648] HybridClock initialized: now 1790246435572378 us; error 27 us; skew 500 ppm
Sep 24 10:40:35 dist-test-slave-j38h krb5kdc[12275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:35.575156 12382 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:40:35.576071 12382 webserver.cc:460] Webserver started at http://127.2.8.1:40921/ using document root <none> and password file <none>
I20260924 10:40:35.576567 12382 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:35.576622 12382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:35.576776 12382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:35.578737 12382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "9a13c15a8a044dd388e8f500be286138"
format_stamp: "Formatted at 2026-09-24 10:40:35 on dist-test-slave-j38h"
server_key: "29685b072b0290d0c09f40da9268e9de"
server_key_iv: "73f8fd1b55a78c040a3776844ddc81a1"
server_key_version: "encryptionkey@0"
I20260924 10:40:35.579222 12382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "9a13c15a8a044dd388e8f500be286138"
format_stamp: "Formatted at 2026-09-24 10:40:35 on dist-test-slave-j38h"
server_key: "29685b072b0290d0c09f40da9268e9de"
server_key_iv: "73f8fd1b55a78c040a3776844ddc81a1"
server_key_version: "encryptionkey@0"
I20260924 10:40:35.582743 12382 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:40:35.585070 12397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:35.586239 12382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:40:35.586351 12382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "9a13c15a8a044dd388e8f500be286138"
format_stamp: "Formatted at 2026-09-24 10:40:35 on dist-test-slave-j38h"
server_key: "29685b072b0290d0c09f40da9268e9de"
server_key_iv: "73f8fd1b55a78c040a3776844ddc81a1"
server_key_version: "encryptionkey@0"
I20260924 10:40:35.586445 12382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:35.597817 12382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:35.601048 12382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:35.601238 12382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:35.601787 12382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:35.602602 12382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:35.602648 12382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:35.602712 12382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:35.602742 12382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:35.612161 12382 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:41833
I20260924 10:40:35.612186 12510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:41833 every 8 connection(s)
I20260924 10:40:35.613054 12382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:40:35.613680 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 12382
I20260924 10:40:35.613840 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:40:35.614208 2080 external_mini_cluster.cc:1428] Setting key 0342712d0128bafaeab56af0b842c3f4
Sep 24 10:40:35 dist-test-slave-j38h krb5kdc[12275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
I20260924 10:40:35.627491 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:35.615024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:36981 (local address 127.2.8.62:43745)
0924 10:40:35.615198 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:35.615211 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:35.616123 (+ 912us) server_negotiation.cc:407] Connection header received
0924 10:40:35.617570 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:35.617573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:35.617624 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:35.617701 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:35.618812 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:35.619339 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:35.620450 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:35.620700 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:35.620808 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:35.623562 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:35.623564 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:35.623566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:35.623590 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:35.625308 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:35.625835 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:35.625838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:35.625839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:35.625931 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:35.626285 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:35.626287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:35.626288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:35.626490 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:35.626642 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:35.627124 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 10:40:35.627290 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:40:35.628273 12513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:35.615426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43745 (local address 127.2.8.1:36981)
0924 10:40:35.615980 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:35.616031 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:35.617369 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:35.617864 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:35.617870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:35.618217 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:35.618669 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:35.618681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:35.619461 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:35.619464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:35.620328 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:35.620336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:35.620796 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:35.620849 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:35.621080 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:35.621681 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:35.621697 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:35.623337 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:35.625433 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:35.625437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:35.625448 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:35.625741 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:35.626050 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:35.626052 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:35.626054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:35.626189 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:35.626628 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:35.626632 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:35.626890 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 10:40:35.627053 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 10:40:35.627413 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":161,"threads_started":1}
I20260924 10:40:35.629657 12511 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43745
I20260924 10:40:35.629920 12511 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:35.630491 12511 heartbeater.cc:511] Master 127.2.8.62:43745 requested a full tablet report, sending...
I20260924 10:40:35.632153 12320 ts_manager.cc:194] Registered new tserver with Master: 9a13c15a8a044dd388e8f500be286138 (127.2.8.1:41833)
I20260924 10:40:35.633445 12320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:36981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:40:35.643627 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43745
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:35.756062 12516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:35.756374 12516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:35.756451 12516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:35.760494 12516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:35.760589 12516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:35.760711 12516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:40:35.765919 12516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:35.766943 12516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:35.767915 12516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:35.777554 12522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:35.777572 12524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:35.777568 12521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:35.777783 12516 server_base.cc:1034] running on GCE node
I20260924 10:40:35.778189 12516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:35.778865 12516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:35.780055 12516 hybrid_clock.cc:648] HybridClock initialized: now 1790246435780039 us; error 30 us; skew 500 ppm
Sep 24 10:40:35 dist-test-slave-j38h krb5kdc[12275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:35.782898 12516 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:40:35.783727 12516 webserver.cc:460] Webserver started at http://127.2.8.2:42939/ using document root <none> and password file <none>
I20260924 10:40:35.784422 12516 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:35.784521 12516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:35.784731 12516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:35.786772 12516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "2504a4fae9ae45ff99261938f7653c92"
format_stamp: "Formatted at 2026-09-24 10:40:35 on dist-test-slave-j38h"
server_key: "8d08f62bb31dd705460f2912c4ea0e5d"
server_key_iv: "cdccc36b27d54051b547edd82343d7a8"
server_key_version: "encryptionkey@0"
I20260924 10:40:35.787321 12516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "2504a4fae9ae45ff99261938f7653c92"
format_stamp: "Formatted at 2026-09-24 10:40:35 on dist-test-slave-j38h"
server_key: "8d08f62bb31dd705460f2912c4ea0e5d"
server_key_iv: "cdccc36b27d54051b547edd82343d7a8"
server_key_version: "encryptionkey@0"
I20260924 10:40:35.791114 12516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:40:35.793463 12531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:35.794695 12516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:40:35.794826 12516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "2504a4fae9ae45ff99261938f7653c92"
format_stamp: "Formatted at 2026-09-24 10:40:35 on dist-test-slave-j38h"
server_key: "8d08f62bb31dd705460f2912c4ea0e5d"
server_key_iv: "cdccc36b27d54051b547edd82343d7a8"
server_key_version: "encryptionkey@0"
I20260924 10:40:35.794934 12516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:35.814158 12516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:35.817231 12516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:35.817427 12516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:35.818001 12516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:35.818840 12516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:35.818908 12516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:35.818981 12516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:35.819033 12516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:35.827397 12516 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:38751
I20260924 10:40:35.827418 12644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:38751 every 8 connection(s)
I20260924 10:40:35.828286 12516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:40:35.829802 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 12516
I20260924 10:40:35.829898 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:40:35.830257 2080 external_mini_cluster.cc:1428] Setting key a722dc019937fd2f6c250338eec02477
Sep 24 10:40:35 dist-test-slave-j38h krb5kdc[12275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
I20260924 10:40:35.844107 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:35.830144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:44125 (local address 127.2.8.62:43745)
0924 10:40:35.830306 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:35.830312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:35.831413 (+ 1101us) server_negotiation.cc:407] Connection header received
0924 10:40:35.833631 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:35.833634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:35.833681 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:35.833820 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:35.835125 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:35.836018 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:35.837398 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:35.837831 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:35.837968 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:35.840398 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:35.840402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:35.840405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:35.840437 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:35.841806 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:35.842371 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:35.842374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:35.842375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:35.842445 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:35.842812 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:35.842819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:35.842823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:35.843079 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:35.843173 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:35.843807 (+ 634us) server_negotiation.cc:299] Negotiation successful
0924 10:40:35.843946 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:40:35.845208 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:35.830581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43745 (local address 127.2.8.2:44125)
0924 10:40:35.831234 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:35.831294 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:35.833362 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:35.834058 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:35.834066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:35.834458 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:35.834967 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:35.834981 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:35.836168 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:35.836171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:35.837272 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:35.837282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:35.837938 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:35.837989 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:35.838138 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:35.838752 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:35.838769 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:35.840229 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:35.841908 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:35.841913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:35.841926 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:35.842288 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:35.842527 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:35.842529 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:35.842530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:35.842718 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:35.843196 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:35.843202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:35.843550 (+ 348us) client_negotiation.cc:769] Sending connection context
0924 10:40:35.843769 (+ 219us) client_negotiation.cc:240] Negotiation successful
0924 10:40:35.844154 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":145,"threads_started":1}
I20260924 10:40:35.846347 12645 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43745
I20260924 10:40:35.846607 12645 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:35.847101 12645 heartbeater.cc:511] Master 127.2.8.62:43745 requested a full tablet report, sending...
I20260924 10:40:35.848163 12320 ts_manager.cc:194] Registered new tserver with Master: 2504a4fae9ae45ff99261938f7653c92 (127.2.8.2:38751)
I20260924 10:40:35.848730 12320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:44125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:40:35.860105 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43745
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:40:35.971653 12650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:40:35.971939 12650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:40:35.972014 12650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:40:35.976212 12650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:40:35.976271 12650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:40:35.976361 12650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:40:35.981653 12650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:40:35.982649 12650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:40:35.983635 12650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:40:35.993284 12655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:40:35.993294 12656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:35.993331 12650 server_base.cc:1034] running on GCE node
W20260924 10:40:35.993294 12658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:40:35.993939 12650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:40:35.994580 12650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:40:35.995761 12650 hybrid_clock.cc:648] HybridClock initialized: now 1790246435995746 us; error 30 us; skew 500 ppm
Sep 24 10:40:35 dist-test-slave-j38h krb5kdc[12275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:40:35.998514 12650 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:40:35.999332 12650 webserver.cc:460] Webserver started at http://127.2.8.3:43551/ using document root <none> and password file <none>
I20260924 10:40:35.999822 12650 fs_manager.cc:359] Metadata directory not provided
I20260924 10:40:35.999878 12650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:40:36.000046 12650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:40:36.002017 12650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "208a7e2d84624aada2eb261232c67521"
format_stamp: "Formatted at 2026-09-24 10:40:35 on dist-test-slave-j38h"
server_key: "2de7fe59b2f59d93a715328157c00cd7"
server_key_iv: "54a1ff991c96c695ea4fcbe9a8e4e570"
server_key_version: "encryptionkey@0"
I20260924 10:40:36.002552 12650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "208a7e2d84624aada2eb261232c67521"
format_stamp: "Formatted at 2026-09-24 10:40:35 on dist-test-slave-j38h"
server_key: "2de7fe59b2f59d93a715328157c00cd7"
server_key_iv: "54a1ff991c96c695ea4fcbe9a8e4e570"
server_key_version: "encryptionkey@0"
I20260924 10:40:36.006089 12650 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:40:36.008246 12665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:36.009466 12650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:40:36.009565 12650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "208a7e2d84624aada2eb261232c67521"
format_stamp: "Formatted at 2026-09-24 10:40:35 on dist-test-slave-j38h"
server_key: "2de7fe59b2f59d93a715328157c00cd7"
server_key_iv: "54a1ff991c96c695ea4fcbe9a8e4e570"
server_key_version: "encryptionkey@0"
I20260924 10:40:36.009645 12650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:40:36.018273 12650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:40:36.021370 12650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:40:36.021510 12650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:40:36.021984 12650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:40:36.022689 12650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:40:36.022732 12650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:36.022768 12650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:40:36.022790 12650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:40:36.030807 12650 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:36877
I20260924 10:40:36.030835 12778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:36877 every 8 connection(s)
I20260924 10:40:36.031651 12650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:40:36.036471 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 12650
I20260924 10:40:36.036582 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:40:36.036871 2080 external_mini_cluster.cc:1428] Setting key 07cdd47398dfb7b98d3f18ab7dea26fd
Sep 24 10:40:36 dist-test-slave-j38h krb5kdc[12275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246435, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:40:36.054320 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:36.033442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:48269 (local address 127.2.8.62:43745)
0924 10:40:36.033575 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:36.033579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:36.034379 (+ 800us) server_negotiation.cc:407] Connection header received
0924 10:40:36.035741 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:36.035745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:36.035801 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:36.035906 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:36.037406 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.038157 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.039442 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.039652 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.039731 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:36.050888 (+ 11157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:36.050892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:36.050894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:36.050919 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:36.052163 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:36.052689 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:36.052694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:36.052696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:36.052777 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:36.053187 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:36.053192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:36.053194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:36.053390 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:36.053496 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:36.054040 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 10:40:36.054160 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:40:36.055183 12781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:36.033792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43745 (local address 127.2.8.3:48269)
0924 10:40:36.034237 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:36.034278 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:36.035535 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:36.036074 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:36.036081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:36.036585 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:40:36.037258 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:36.037274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.038297 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.038301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:36.039331 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:36.039341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.039735 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.039775 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:36.039911 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:36.048753 (+ 8842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:40:36.048772 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:40:36.050707 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:40:36.052287 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:36.052293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:36.052310 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:36.052582 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:36.052871 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:40:36.052875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:40:36.052877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:40:36.053091 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:40:36.053507 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:40:36.053524 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:40:36.053789 (+ 265us) client_negotiation.cc:769] Sending connection context
0924 10:40:36.053979 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 10:40:36.054320 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":114,"threads_started":1}
I20260924 10:40:36.056205 12779 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43745
I20260924 10:40:36.056422 12779 heartbeater.cc:464] Registering TS with master...
I20260924 10:40:36.056861 12779 heartbeater.cc:511] Master 127.2.8.62:43745 requested a full tablet report, sending...
I20260924 10:40:36.057862 12320 ts_manager.cc:194] Registered new tserver with Master: 208a7e2d84624aada2eb261232c67521 (127.2.8.3:36877)
I20260924 10:40:36.058393 12320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:48269
I20260924 10:40:36.062476 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:40:36.070953 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:36.064323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39412 (local address 127.2.8.62:43745)
0924 10:40:36.064436 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:36.064440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:36.064595 (+ 155us) server_negotiation.cc:407] Connection header received
0924 10:40:36.064687 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:36.064690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:36.064726 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:36.064791 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:40:36.065453 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.065935 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.066703 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.066899 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.066972 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:36.068267 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:40:36.068269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:40:36.068271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:40:36.068287 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:40:36.069506 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:36.069861 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:36.069863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:36.069864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:36.069932 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:40:36.070148 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:40:36.070150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:40:36.070151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:40:36.070467 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:40:36.070574 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:36.070758 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 10:40:36.070843 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:40:36.073886 12320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:40:36.076786 12320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:40:36.090273 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:36.084953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.2.8.2:38751)
0924 10:40:36.085300 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:36.085314 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:36.085339 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:40:36.085416 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:36.085422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:36.085653 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:36.085808 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:36.086292 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.087385 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.088758 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.089535 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.089671 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:36.089816 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:36.090027 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 10:40:36.090115 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":125,"threads_started":1}
I20260924 10:40:36.090402 12790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:36.084697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:38751 (local address 127.0.0.1:58266)
0924 10:40:36.085035 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:36.085079 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:36.085196 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:36.085997 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:36.086002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:36.086025 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:36.086155 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:36.086162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.087510 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.087516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:36.088629 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:36.088638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.089671 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.089675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:36.089824 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:36.089872 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 10:40:36.089997 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 10:40:36.090112 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260924 10:40:36.091028 12793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:36.085393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:36877 (local address 127.0.0.1:58696)
0924 10:40:36.086150 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:36.086164 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:36.086275 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:36.086696 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:36.086699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:36.086716 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:36.086809 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:36.086819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.088169 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.088173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:36.089394 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:36.089404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.090638 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.090641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:36.090735 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:36.090755 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:40:36.090827 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 10:40:36.090889 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":90,"threads_started":1}
I20260924 10:40:36.091091 12794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:36.085714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.2.8.3:36877)
0924 10:40:36.086173 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:36.086188 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:36.086242 (+ 54us) server_negotiation.cc:407] Connection header received
0924 10:40:36.086416 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:36.086420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:36.086561 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:36.086710 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:36.086929 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.087997 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.089523 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.090340 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.090443 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:36.090579 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:36.090881 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 10:40:36.090966 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":107,"threads_started":1}
I20260924 10:40:36.091416 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:36.085820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.2.8.1:41833)
0924 10:40:36.086180 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:36.086195 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:36.086494 (+ 299us) server_negotiation.cc:407] Connection header received
0924 10:40:36.086567 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:36.086572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:36.086741 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:36.086869 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:36.087219 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.088273 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.089718 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.090457 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.090555 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:36.090665 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:36.091155 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 10:40:36.091261 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":79,"threads_started":1}
I20260924 10:40:36.091064 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:36.085451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:41833 (local address 127.0.0.1:35090)
0924 10:40:36.086312 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:36.086327 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:36.086410 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:36.087011 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:36.087014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:36.087030 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:36.087102 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:36.087107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.088403 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.088406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:36.089546 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:36.089554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.090752 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.090756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:36.090842 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:36.090859 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:40:36.090915 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 10:40:36.090967 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":87,"threads_started":1}
I20260924 10:40:36.093516 12579 tablet_service.cc:1467] Processing CreateTablet for tablet 1b5dbdfe59324a3b822c6e9d598c2d26 (DEFAULT_TABLE table=test-table [id=130e1db85a2844eb9af2b3d82b0b8522]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:36.093819 12713 tablet_service.cc:1467] Processing CreateTablet for tablet 1b5dbdfe59324a3b822c6e9d598c2d26 (DEFAULT_TABLE table=test-table [id=130e1db85a2844eb9af2b3d82b0b8522]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:36.094069 12445 tablet_service.cc:1467] Processing CreateTablet for tablet 1b5dbdfe59324a3b822c6e9d598c2d26 (DEFAULT_TABLE table=test-table [id=130e1db85a2844eb9af2b3d82b0b8522]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:40:36.094622 12579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b5dbdfe59324a3b822c6e9d598c2d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:36.094627 12713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b5dbdfe59324a3b822c6e9d598c2d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:36.094944 12445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b5dbdfe59324a3b822c6e9d598c2d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:40:36.101575 12796 tablet_bootstrap.cc:492] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 9a13c15a8a044dd388e8f500be286138: Bootstrap starting.
I20260924 10:40:36.102138 12798 tablet_bootstrap.cc:492] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92: Bootstrap starting.
I20260924 10:40:36.102276 12797 tablet_bootstrap.cc:492] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521: Bootstrap starting.
I20260924 10:40:36.103598 12796 tablet_bootstrap.cc:654] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 9a13c15a8a044dd388e8f500be286138: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:36.104206 12797 tablet_bootstrap.cc:654] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:36.104206 12798 tablet_bootstrap.cc:654] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92: Neither blocks nor log segments found. Creating new log.
I20260924 10:40:36.104710 12796 log.cc:824] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 9a13c15a8a044dd388e8f500be286138: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:36.105535 12798 log.cc:824] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:36.105535 12797 log.cc:824] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521: Log is configured to *not* fsync() on all Append() calls
I20260924 10:40:36.106873 12796 tablet_bootstrap.cc:492] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 9a13c15a8a044dd388e8f500be286138: No bootstrap required, opened a new log
I20260924 10:40:36.107082 12796 ts_tablet_manager.cc:1397] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 9a13c15a8a044dd388e8f500be286138: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:40:36.107332 12798 tablet_bootstrap.cc:492] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92: No bootstrap required, opened a new log
I20260924 10:40:36.107332 12797 tablet_bootstrap.cc:492] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521: No bootstrap required, opened a new log
I20260924 10:40:36.107509 12797 ts_tablet_manager.cc:1397] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:40:36.107509 12798 ts_tablet_manager.cc:1397] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 10:40:36.110025 12796 raft_consensus.cc:348] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 9a13c15a8a044dd388e8f500be286138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2504a4fae9ae45ff99261938f7653c92" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 38751 } } peers { permanent_uuid: "208a7e2d84624aada2eb261232c67521" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36877 } } peers { permanent_uuid: "9a13c15a8a044dd388e8f500be286138" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41833 } }
I20260924 10:40:36.110263 12796 raft_consensus.cc:374] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 9a13c15a8a044dd388e8f500be286138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:36.110337 12796 raft_consensus.cc:729] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 9a13c15a8a044dd388e8f500be286138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a13c15a8a044dd388e8f500be286138, State: Initialized, Role: FOLLOWER
I20260924 10:40:36.110803 12796 consensus_queue.cc:260] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 9a13c15a8a044dd388e8f500be286138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2504a4fae9ae45ff99261938f7653c92" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 38751 } } peers { permanent_uuid: "208a7e2d84624aada2eb261232c67521" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36877 } } peers { permanent_uuid: "9a13c15a8a044dd388e8f500be286138" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41833 } }
I20260924 10:40:36.111215 12798 raft_consensus.cc:348] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2504a4fae9ae45ff99261938f7653c92" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 38751 } } peers { permanent_uuid: "208a7e2d84624aada2eb261232c67521" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36877 } } peers { permanent_uuid: "9a13c15a8a044dd388e8f500be286138" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41833 } }
I20260924 10:40:36.111214 12797 raft_consensus.cc:348] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2504a4fae9ae45ff99261938f7653c92" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 38751 } } peers { permanent_uuid: "208a7e2d84624aada2eb261232c67521" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36877 } } peers { permanent_uuid: "9a13c15a8a044dd388e8f500be286138" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41833 } }
I20260924 10:40:36.111450 12797 raft_consensus.cc:374] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:36.111455 12511 heartbeater.cc:503] Master 127.2.8.62:43745 was elected leader, sending a full tablet report...
I20260924 10:40:36.111450 12798 raft_consensus.cc:374] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:40:36.111518 12798 raft_consensus.cc:729] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2504a4fae9ae45ff99261938f7653c92, State: Initialized, Role: FOLLOWER
I20260924 10:40:36.111562 12797 raft_consensus.cc:729] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 208a7e2d84624aada2eb261232c67521, State: Initialized, Role: FOLLOWER
I20260924 10:40:36.111943 12797 consensus_queue.cc:260] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2504a4fae9ae45ff99261938f7653c92" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 38751 } } peers { permanent_uuid: "208a7e2d84624aada2eb261232c67521" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36877 } } peers { permanent_uuid: "9a13c15a8a044dd388e8f500be286138" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41833 } }
I20260924 10:40:36.111943 12798 consensus_queue.cc:260] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2504a4fae9ae45ff99261938f7653c92" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 38751 } } peers { permanent_uuid: "208a7e2d84624aada2eb261232c67521" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36877 } } peers { permanent_uuid: "9a13c15a8a044dd388e8f500be286138" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41833 } }
I20260924 10:40:36.112895 12797 ts_tablet_manager.cc:1428] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:40:36.112921 12798 ts_tablet_manager.cc:1428] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260924 10:40:36.113251 12645 heartbeater.cc:503] Master 127.2.8.62:43745 was elected leader, sending a full tablet report...
I20260924 10:40:36.114138 12779 heartbeater.cc:503] Master 127.2.8.62:43745 was elected leader, sending a full tablet report...
I20260924 10:40:36.115300 12796 ts_tablet_manager.cc:1428] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 9a13c15a8a044dd388e8f500be286138: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260924 10:40:36.235533 12803 raft_consensus.cc:482] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:40:36.235757 12803 raft_consensus.cc:504] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2504a4fae9ae45ff99261938f7653c92" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 38751 } } peers { permanent_uuid: "208a7e2d84624aada2eb261232c67521" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36877 } } peers { permanent_uuid: "9a13c15a8a044dd388e8f500be286138" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41833 } }
I20260924 10:40:36.236900 12803 leader_election.cc:288] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 208a7e2d84624aada2eb261232c67521 (127.2.8.3:36877), 9a13c15a8a044dd388e8f500be286138 (127.2.8.1:41833)
I20260924 10:40:36.240525 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:36.237258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:49753 (local address 127.2.8.1:41833)
0924 10:40:36.237381 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:36.237385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:36.237471 (+ 86us) server_negotiation.cc:407] Connection header received
0924 10:40:36.237654 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:36.237657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:36.237700 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:36.237785 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:36.238054 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.238547 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.239570 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.239953 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.240084 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:36.240150 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:36.240327 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 10:40:36.240395 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:40:36.240517 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:36.237177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:41833 (local address 127.2.8.2:49753)
0924 10:40:36.237381 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:36.237398 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:36.237498 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:36.237797 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:36.237799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:36.237814 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:36.237932 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:36.237937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.238658 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.238663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:36.239436 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:36.239444 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.240090 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.240094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:36.240197 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:36.240223 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 10:40:36.240311 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 10:40:36.240385 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260924 10:40:36.240649 12794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:36.237370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:44193 (local address 127.2.8.3:36877)
0924 10:40:36.237502 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:36.237506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:36.237671 (+ 165us) server_negotiation.cc:407] Connection header received
0924 10:40:36.237819 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:36.237821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:36.237872 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:36.237955 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:40:36.238206 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.238687 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.239688 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.240102 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.240212 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:36.240269 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:36.240458 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 10:40:36.240531 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:40:36.240671 12805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:36.237177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:36877 (local address 127.2.8.2:44193)
0924 10:40:36.237578 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:36.237589 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:40:36.237708 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:40:36.237962 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:40:36.237965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:40:36.237986 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:40:36.238101 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:36.238106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.238789 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.238795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:36.239567 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:40:36.239575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.240227 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.240231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:40:36.240333 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:36.240361 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:40:36.240444 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 10:40:36.240510 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":29,"thread_start_us":129,"threads_started":1}
I20260924 10:40:36.241238 12465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5dbdfe59324a3b822c6e9d598c2d26" candidate_uuid: "2504a4fae9ae45ff99261938f7653c92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a13c15a8a044dd388e8f500be286138" is_pre_election: true
I20260924 10:40:36.241240 12733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5dbdfe59324a3b822c6e9d598c2d26" candidate_uuid: "2504a4fae9ae45ff99261938f7653c92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "208a7e2d84624aada2eb261232c67521" is_pre_election: true
I20260924 10:40:36.241560 12465 raft_consensus.cc:2454] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 9a13c15a8a044dd388e8f500be286138 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2504a4fae9ae45ff99261938f7653c92 in term 0.
I20260924 10:40:36.241560 12733 raft_consensus.cc:2454] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2504a4fae9ae45ff99261938f7653c92 in term 0.
I20260924 10:40:36.242050 12535 leader_election.cc:302] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 208a7e2d84624aada2eb261232c67521, 2504a4fae9ae45ff99261938f7653c92; no voters:
I20260924 10:40:36.242329 12803 raft_consensus.cc:2781] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:40:36.242422 12803 raft_consensus.cc:482] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:40:36.242478 12803 raft_consensus.cc:3037] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:36.243476 12803 raft_consensus.cc:504] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2504a4fae9ae45ff99261938f7653c92" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 38751 } } peers { permanent_uuid: "208a7e2d84624aada2eb261232c67521" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36877 } } peers { permanent_uuid: "9a13c15a8a044dd388e8f500be286138" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41833 } }
I20260924 10:40:36.243814 12803 leader_election.cc:288] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [CANDIDATE]: Term 1 election: Requested vote from peers 208a7e2d84624aada2eb261232c67521 (127.2.8.3:36877), 9a13c15a8a044dd388e8f500be286138 (127.2.8.1:41833)
I20260924 10:40:36.244177 12733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5dbdfe59324a3b822c6e9d598c2d26" candidate_uuid: "2504a4fae9ae45ff99261938f7653c92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "208a7e2d84624aada2eb261232c67521"
I20260924 10:40:36.244187 12465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b5dbdfe59324a3b822c6e9d598c2d26" candidate_uuid: "2504a4fae9ae45ff99261938f7653c92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a13c15a8a044dd388e8f500be286138"
I20260924 10:40:36.244309 12733 raft_consensus.cc:3037] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:36.244318 12465 raft_consensus.cc:3037] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 9a13c15a8a044dd388e8f500be286138 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:40:36.245366 12733 raft_consensus.cc:2454] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2504a4fae9ae45ff99261938f7653c92 in term 1.
I20260924 10:40:36.245417 12465 raft_consensus.cc:2454] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 9a13c15a8a044dd388e8f500be286138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2504a4fae9ae45ff99261938f7653c92 in term 1.
I20260924 10:40:36.245642 12535 leader_election.cc:302] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 208a7e2d84624aada2eb261232c67521, 2504a4fae9ae45ff99261938f7653c92; no voters:
I20260924 10:40:36.245803 12803 raft_consensus.cc:2781] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:40:36.246035 12803 raft_consensus.cc:686] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [term 1 LEADER]: Becoming Leader. State: Replica: 2504a4fae9ae45ff99261938f7653c92, State: Running, Role: LEADER
I20260924 10:40:36.246376 12803 consensus_queue.cc:237] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2504a4fae9ae45ff99261938f7653c92" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 38751 } } peers { permanent_uuid: "208a7e2d84624aada2eb261232c67521" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36877 } } peers { permanent_uuid: "9a13c15a8a044dd388e8f500be286138" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41833 } }
I20260924 10:40:36.249385 12320 catalog_manager.cc:5495] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2504a4fae9ae45ff99261938f7653c92 (127.2.8.2). New cstate: current_term: 1 leader_uuid: "2504a4fae9ae45ff99261938f7653c92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2504a4fae9ae45ff99261938f7653c92" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 38751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "208a7e2d84624aada2eb261232c67521" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a13c15a8a044dd388e8f500be286138" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41833 } health_report { overall_health: UNKNOWN } } }
I20260924 10:40:36.277894 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:36.274561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.2.8.2:38751)
0924 10:40:36.274700 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:40:36.274704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:40:36.274718 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:40:36.274815 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:40:36.274818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:40:36.274871 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:40:36.274968 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:40:36.275177 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.275687 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.276519 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:40:36.276715 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:40:36.276789 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:40:36.277050 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:40:36.277454 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:40:36.277572 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:40:36.277632 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 10:40:36.277679 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 10:40:36.282578 12780 tablet.cc:2369] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:40:36.283552 12465 raft_consensus.cc:1261] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 9a13c15a8a044dd388e8f500be286138 [term 1 FOLLOWER]: Refusing update from remote peer 2504a4fae9ae45ff99261938f7653c92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:36.283699 12733 raft_consensus.cc:1261] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521 [term 1 FOLLOWER]: Refusing update from remote peer 2504a4fae9ae45ff99261938f7653c92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:40:36.284353 12803 consensus_queue.cc:1035] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a13c15a8a044dd388e8f500be286138" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:36.284541 12806 consensus_queue.cc:1035] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "208a7e2d84624aada2eb261232c67521" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:40:36.300863 12320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:40:36.301072 12320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:40:36.303324 12320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1087361dc2bb4b75b51c86331189b8ea: Sending DeleteTablet for 3 replicas of tablet 1b5dbdfe59324a3b822c6e9d598c2d26
I20260924 10:40:36.304129 12445 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b5dbdfe59324a3b822c6e9d598c2d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:36 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:35090
I20260924 10:40:36.304129 12713 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b5dbdfe59324a3b822c6e9d598c2d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:36 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:58696
I20260924 10:40:36.304168 12579 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b5dbdfe59324a3b822c6e9d598c2d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:40:36 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:58266
I20260924 10:40:36.304601 12818 tablet_replica.cc:330] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521: stopping tablet replica
I20260924 10:40:36.304898 12818 raft_consensus.cc:2229] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:40:36.305171 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 12382
I20260924 10:40:36.305239 12818 raft_consensus.cc:2258] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:36.306751 12818 ts_tablet_manager.cc:1905] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:36.309408 12820 tablet_replica.cc:330] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92: stopping tablet replica
I20260924 10:40:36.309737 12820 raft_consensus.cc:2229] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:40:36.310158 12818 ts_tablet_manager.cc:1918] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:40:36.310169 12820 raft_consensus.cc:2258] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:40:36.310317 12818 log.cc:1196] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/1b5dbdfe59324a3b822c6e9d598c2d26
I20260924 10:40:36.310837 12818 ts_tablet_manager.cc:1939] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 208a7e2d84624aada2eb261232c67521: Deleting consensus metadata
I20260924 10:40:36.311463 12820 ts_tablet_manager.cc:1905] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:40:36.312203 12307 catalog_manager.cc:4841] TS 208a7e2d84624aada2eb261232c67521 (127.2.8.3:36877): tablet 1b5dbdfe59324a3b822c6e9d598c2d26 (table test-table [id=130e1db85a2844eb9af2b3d82b0b8522]) successfully deleted
I20260924 10:40:36.314023 12820 ts_tablet_manager.cc:1918] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:40:36.314139 12820 log.cc:1196] T 1b5dbdfe59324a3b822c6e9d598c2d26 P 2504a4fae9ae45ff99261938f7653c92: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/1b5dbdfe59324a3b822c6e9d598c2d26
I20260924 10:40:36.314380 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 12516
W20260924 10:40:36.314376 12306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.8.1:41833 (error 108)
I20260924 10:40:36.316793 12793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:36.316508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:41833 (local address 127.0.0.1:35098)
0924 10:40:36.316649 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:36.316716 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:41833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 10:40:36.317068 12306 catalog_manager.cc:4584] TS 9a13c15a8a044dd388e8f500be286138 (127.2.8.1:41833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b5dbdfe59324a3b822c6e9d598c2d26: Network error: Client connection negotiation failed: client connection to 127.2.8.1:41833: connect: Connection refused (error 111)
I20260924 10:40:36.320410 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 12650
I20260924 10:40:36.321146 12793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:40:36.320615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:38751 (local address 127.0.0.1:58290)
0924 10:40:36.320967 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 10:40:36.321050 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.2:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282}
I20260924 10:40:36.325968 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 12288
2026-09-24T10:40:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2345 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:40:36 dist-test-slave-j38h krb5kdc[12824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:40:36 dist-test-slave-j38h krb5kdc[12824](info): set up 2 sockets
Sep 24 10:40:36 dist-test-slave-j38h krb5kdc[12824](info): commencing operation
krb5kdc: starting...
W20260924 10:40:37.410066 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:40:37 dist-test-slave-j38h krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:40:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:40:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.8.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.8.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.8.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/rangeradmin_127.2.8.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.8.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/rangeradmin_127.2.8.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.8.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.8.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.8.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/rangerlookup_127.2.8.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.8.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/rangerlookup_127.2.8.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.8.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.8.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.8.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/HTTP_127.2.8.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.8.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/HTTP_127.2.8.20@KRBTEST.COM.keytab.
I20260924 10:40:37.574035 2080 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2SD_Oz/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/postgres -l logfile start
2026-09-24 10:40:40.137 UTC [12854] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 10:40:40.137 UTC [12854] LOG: listening on IPv4 address "127.2.8.20", port 50331
2026-09-24 10:40:40.140 UTC [12854] LOG: listening on Unix socket "/tmp/.s.PGSQL.50331"
2026-09-24 10:40:40.161 UTC [12857] LOG: database system was shut down at 2026-09-24 10:40:38 UTC
2026-09-24 10:40:40.165 UTC [12854] LOG: database system is ready to accept connections
I20260924 10:40:41.133769 2080 mini_postgres.cc:96] Postgres bound to 50331
2026-09-24 10:40:41.140 UTC [12865] FATAL: database "slave" does not exist
127.2.8.20:50331 - accepting connections
I20260924 10:40:41.140818 2080 mini_ranger.cc:161] Starting Ranger...
I20260924 10:40:41.156625 2080 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 10:40:41.648010 2080 mini_ranger.cc:87] Created ranger Postgres database
I20260924 10:40:41.648123 2080 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-admin
2026-09-24 10:40:41,936 [I] DB FLAVOR :POSTGRES
2026-09-24 10:40:41,936 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:40:41,936 [I] Checking connection..
2026-09-24 10:40:41,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 10:40:42,174 [I] Checking connection passed.
2026-09-24 10:40:42,174 [I] --------- Verifying version history table ---------
2026-09-24 10:40:42,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:40:42,429 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:40:42,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:40:42,658 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:40:42,658 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 10:40:42,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 10:40:42.921 UTC [12929] WARNING: there is no transaction in progress
2026-09-24 10:40:42,927 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 10:40:42,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:40:43,157 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 10:40:43,157 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:40:43,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 10:40:43,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 10:40:43,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j38h.c.gcp-upstream.internal','N') ;"
2026-09-24 10:40:43,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:40:44,110 [I] Table x_portal_user does not exist in database ranger
2026-09-24 10:40:44,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:40:44,339 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 10:40:44,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:40:44,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:40:44,820 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 10:40:44,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 10:40:45.768 UTC [13037] WARNING: there is no transaction in progress
2026-09-24 10:40:45.784 UTC [13037] WARNING: there is no transaction in progress
2026-09-24 10:40:45.801 UTC [13037] WARNING: there is no transaction in progress
2026-09-24 10:40:45.817 UTC [13037] WARNING: there is no transaction in progress
2026-09-24 10:40:45.833 UTC [13037] WARNING: there is no transaction in progress
2026-09-24 10:40:45.851 UTC [13037] WARNING: there is no transaction in progress
2026-09-24 10:40:45.905 UTC [13037] WARNING: there is no transaction in progress
2026-09-24 10:40:45.912 UTC [13037] WARNING: there is no transaction in progress
2026-09-24 10:40:45.927 UTC [13037] WARNING: there is no transaction in progress
2026-09-24 10:40:45.938 UTC [13037] WARNING: there is no transaction in progress
2026-09-24 10:40:45.948 UTC [13037] WARNING: there is no transaction in progress
2026-09-24 10:40:45.962 UTC [13037] WARNING: there is no transaction in progress
2026-09-24 10:40:45.972 UTC [13037] WARNING: there is no transaction in progress
2026-09-24 10:40:45.980 UTC [13037] WARNING: there is no transaction in progress
2026-09-24 10:40:45.987 UTC [13037] WARNING: there is no transaction in progress
2026-09-24 10:40:45.994 UTC [13037] WARNING: there is no transaction in progress
2026-09-24 10:40:46,649 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 10:40:46,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 10:40:46,883 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 10:40:46,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:40:47,127 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 10:40:47,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:40:47,389 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 10:40:47,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:40:47,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:40:47,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j38h.c.gcp-upstream.internal';"
2026-09-24 10:40:48,100 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 10:40:48,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 10:40:48,357 [I] DB_PATCHES have already been applied
I20260924 10:40:48.363040 2080 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-admin:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2SD_Oz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 10:40:48.363143 2080 mini_ranger.cc:193] Using host: 127.2.8.20
I20260924 10:40:48.366703 2080 mini_ranger.cc:227] Ranger admin URL: http://127.2.8.20:50395
Sep 24, 2026 10:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 10:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 10:40:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 10:40:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 10:40:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 10:40:49.472702 2080 mini_ranger.cc:160] Time spent starting Ranger: real 8.332s user 0.003s sys 0.007s
Sep 24, 2026 10:41:10 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 10:41:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 10:41:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 10:41:10 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 10:41:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 10:41:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 10:41:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 10:41:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 10:41:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 10:41:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 10:41:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 10:41:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 10:41:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 10:41:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 10:41:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 10:41:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 10:41:10 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 10:41:11 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 10:41:11.840838 2080 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.8.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.8.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.8.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/rangerkms_127.2.8.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.8.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/rangerkms_127.2.8.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.8.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.8.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.8.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/HTTP_127.2.8.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.8.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/HTTP_127.2.8.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 10:41:11 dist-test-slave-j38h krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246471, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 10:41:11.930905 2080 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 10:41:11.940762 2080 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 10:41:12.863381 2080 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 10:41:12.863466 2080 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-kms
I20260924 10:41:12.863579 2080 mini_ranger_kms.cc:227] Using postgres at 127.2.8.20:50331
2026-09-24 10:41:13,057 [I] DB FLAVOR :POSTGRES
2026-09-24 10:41:13,058 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:41:13,058 [I] Checking connection
2026-09-24 10:41:13,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 10:41:13,297 [I] connection success
2026-09-24 10:41:13,297 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 10:41:13,297 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 10:41:13,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 10:41:13,532 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 10:41:13,532 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:41:13,533 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 10:41:13,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:50331/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 10:41:13,800 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 10:41:14.259855 2080 mini_ranger_kms.cc:310] Created kms service
I20260924 10:41:14.455545 2080 mini_ranger_kms.cc:326] Created kudu user
I20260924 10:41:14.498519 2080 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 10:41:14 dist-test-slave-j38h krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246471, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:41:14.918009 2080 mini_ranger_kms.cc:384] Added ranger policy
I20260924 10:41:14.918311 2080 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-kms:/tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 10:41:14.918362 2080 mini_ranger_kms.cc:240] Using host: 127.2.8.20
I20260924 10:41:14.922258 2080 mini_ranger_kms.cc:276] Ranger KMS PID: 13217
I20260924 10:41:14.922394 2080 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.8.20:55927
Sep 24, 2026 10:41:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:41:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:41:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 10:41:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 10:41:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 10:41:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 10:41:17.207019 2080 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.276s user 0.009s sys 0.169s
I20260924 10:41:17.207173 2080 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 10:41:17.207305 2080 mini_ranger_kms.cc:401] 127.2.8.20:55927/kms/v1/keys
Sep 24 10:41:17 dist-test-slave-j38h krb5kdc[12824](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790246477, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.8.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 10:41:17 dist-test-slave-j38h krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246477, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.8.20@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
Sep 24 10:41:18 dist-test-slave-j38h krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246477, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.8.20@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
Sep 24 10:41:20 dist-test-slave-j38h krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:41:20.392145 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:44203
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:44203
--ranger_config_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.8.20:55927/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:41:20.504134 13268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:41:20.504393 13268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:41:20.504431 13268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:41:20.508551 13268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:41:20.508610 13268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:41:20.508628 13268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:41:20.508644 13268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:41:20.508659 13268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:41:20.514035 13268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.8.20:55927/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:44203
--ranger_config_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:44203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:41:20.515234 13268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:41:20.516269 13268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:41:20.525205 13276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:41:20.525266 13274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:41:20.525346 13268 server_base.cc:1034] running on GCE node
W20260924 10:41:20.525452 13273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:41:20.526038 13268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:41:20.527175 13268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:41:20.528352 13268 hybrid_clock.cc:648] HybridClock initialized: now 1790246480528329 us; error 46 us; skew 500 ppm
Sep 24 10:41:20 dist-test-slave-j38h krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246480, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:41:20.534982 13268 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:41:20.535887 13268 webserver.cc:460] Webserver started at http://127.2.8.62:37665/ using document root <none> and password file <none>
I20260924 10:41:20.536398 13268 fs_manager.cc:359] Metadata directory not provided
I20260924 10:41:20.536450 13268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:41:20.536610 13268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:41:20 dist-test-slave-j38h krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246480, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:41:20.591250 13268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "8fde9a21bf6d40428deea067c37315a3"
format_stamp: "Formatted at 2026-09-24 10:41:20 on dist-test-slave-j38h"
server_key: "2ad691b1248b7e66ce87db4ec814e19d"
server_key_iv: "0edcad6703fe53181156dd02dc6b8001"
server_key_version: "kuduclusterkey@0"
I20260924 10:41:20.592003 13268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "8fde9a21bf6d40428deea067c37315a3"
format_stamp: "Formatted at 2026-09-24 10:41:20 on dist-test-slave-j38h"
server_key: "2ad691b1248b7e66ce87db4ec814e19d"
server_key_iv: "0edcad6703fe53181156dd02dc6b8001"
server_key_version: "kuduclusterkey@0"
I20260924 10:41:20.596417 13268 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 10:41:20.614112 13283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:41:20.615660 13268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:41:20.615823 13268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "8fde9a21bf6d40428deea067c37315a3"
format_stamp: "Formatted at 2026-09-24 10:41:20 on dist-test-slave-j38h"
server_key: "2ad691b1248b7e66ce87db4ec814e19d"
server_key_iv: "0edcad6703fe53181156dd02dc6b8001"
server_key_version: "kuduclusterkey@0"
I20260924 10:41:20.615960 13268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:41:20.634497 13268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:41:20.637774 13268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:41:20.637979 13268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:41:20.645287 13268 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:44203
I20260924 10:41:20.645300 13335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:44203 every 8 connection(s)
I20260924 10:41:20.646216 13268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:41:20.649036 13336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:41:20.650657 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 13268
I20260924 10:41:20.650769 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:41:20.651144 2080 external_mini_cluster.cc:1428] Setting key 00fcbb9b0ea1544ce4adf164e23ecbb7
I20260924 10:41:20.656533 13336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3: Bootstrap starting.
I20260924 10:41:20.659350 13336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3: Neither blocks nor log segments found. Creating new log.
I20260924 10:41:20.660413 13336 log.cc:824] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:41:20 dist-test-slave-j38h krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:41:20.663005 13336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3: No bootstrap required, opened a new log
I20260924 10:41:20.667618 13336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fde9a21bf6d40428deea067c37315a3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44203 } }
I20260924 10:41:20.667897 13336 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:41:20.667969 13336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fde9a21bf6d40428deea067c37315a3, State: Initialized, Role: FOLLOWER
I20260924 10:41:20.668438 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:20.652639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.2.8.62:44203)
0924 10:41:20.653119 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 10:41:20.653139 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 10:41:20.653179 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:41:20.654408 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:41:20.654431 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:41:20.654812 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:41:20.655130 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:41:20.656197 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:20.657382 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:20.658702 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:20.659167 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:20.659313 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:20.662467 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:41:20.662473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:41:20.662492 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:41:20.662524 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:41:20.665265 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:41:20.665745 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:41:20.665752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:41:20.665759 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:41:20.665896 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:41:20.666161 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:41:20.666165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:41:20.666167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:41:20.666703 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:41:20.666914 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 10:41:20.667149 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 10:41:20.667441 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":150,"threads_started":1}
I20260924 10:41:20.668467 13336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fde9a21bf6d40428deea067c37315a3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44203 } }
I20260924 10:41:20.668622 13336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:41:20.668699 13336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:41:20.668803 13336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:41:20.669991 13336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fde9a21bf6d40428deea067c37315a3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44203 } }
I20260924 10:41:20.670361 13336 leader_election.cc:302] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fde9a21bf6d40428deea067c37315a3; no voters:
I20260924 10:41:20.670737 13336 leader_election.cc:288] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:41:20.670845 13341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:41:20.671068 13341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [term 1 LEADER]: Becoming Leader. State: Replica: 8fde9a21bf6d40428deea067c37315a3, State: Running, Role: LEADER
I20260924 10:41:20.671355 13341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fde9a21bf6d40428deea067c37315a3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44203 } }
I20260924 10:41:20.672020 13336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:41:20.673147 13343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fde9a21bf6d40428deea067c37315a3. Latest consensus state: current_term: 1 leader_uuid: "8fde9a21bf6d40428deea067c37315a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fde9a21bf6d40428deea067c37315a3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44203 } } }
I20260924 10:41:20.673288 13343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [sys.catalog]: This master's current role is: LEADER
I20260924 10:41:20.673547 13342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fde9a21bf6d40428deea067c37315a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fde9a21bf6d40428deea067c37315a3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44203 } } }
I20260924 10:41:20.673640 13342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3 [sys.catalog]: This master's current role is: LEADER
I20260924 10:41:20.673897 13350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:41:20.677349 13350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:41:20.677825 13336 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 10:41:20.687988 13350 catalog_manager.cc:1347] Generated new cluster ID: 485b43c31f984cf19329a30330cd452b
I20260924 10:41:20.688088 13350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:41:20.704948 13350 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:41:20.706344 13350 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:41:20.714742 13350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3: Generated new TSK 0
I20260924 10:41:20.715543 13350 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:41:21.465201 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.8.20:55927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:44203
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:41:21.612249 13381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:41:21.612531 13381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:41:21.612650 13381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:41:21.616755 13381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:41:21.616832 13381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:41:21.616922 13381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:41:21.622390 13381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.8.20:55927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:41:21.623410 13381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:41:21.624406 13381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:41:21.635394 13386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:41:21.635408 13387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:41:21.636292 13389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:41:21.636734 13381 server_base.cc:1034] running on GCE node
I20260924 10:41:21.637236 13381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:41:21.637976 13381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:41:21.639163 13381 hybrid_clock.cc:648] HybridClock initialized: now 1790246481639145 us; error 37 us; skew 500 ppm
Sep 24 10:41:21 dist-test-slave-j38h krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:41:21.642239 13381 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:41:21.643337 13381 webserver.cc:460] Webserver started at http://127.2.8.1:41711/ using document root <none> and password file <none>
I20260924 10:41:21.644026 13381 fs_manager.cc:359] Metadata directory not provided
I20260924 10:41:21.644122 13381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:41:21.644395 13381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:41:21 dist-test-slave-j38h krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:41:21.666435 13381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "4f2ee7e1ab354f6db28fef54743e0a88"
format_stamp: "Formatted at 2026-09-24 10:41:21 on dist-test-slave-j38h"
server_key: "1b7e03f94b547f61523b5fa522edcc61"
server_key_iv: "96dfa0603ea0bf01f063967c78853e93"
server_key_version: "kuduclusterkey@0"
I20260924 10:41:21.672389 13381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "4f2ee7e1ab354f6db28fef54743e0a88"
format_stamp: "Formatted at 2026-09-24 10:41:21 on dist-test-slave-j38h"
server_key: "1b7e03f94b547f61523b5fa522edcc61"
server_key_iv: "96dfa0603ea0bf01f063967c78853e93"
server_key_version: "kuduclusterkey@0"
I20260924 10:41:21.677920 13381 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:41:21.693493 13398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:41:21.694977 13381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:41:21.695170 13381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "4f2ee7e1ab354f6db28fef54743e0a88"
format_stamp: "Formatted at 2026-09-24 10:41:21 on dist-test-slave-j38h"
server_key: "1b7e03f94b547f61523b5fa522edcc61"
server_key_iv: "96dfa0603ea0bf01f063967c78853e93"
server_key_version: "kuduclusterkey@0"
I20260924 10:41:21.695393 13381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:41:21.709335 13381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:41:21.713006 13381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:41:21.713315 13381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:41:21.714048 13381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:41:21.715337 13381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:41:21.715441 13381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:41:21.715561 13381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:41:21.715634 13381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:41:21.730172 13381 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:33513
I20260924 10:41:21.731436 13381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:41:21.732602 13511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:33513 every 8 connection(s)
I20260924 10:41:21.738422 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 13381
I20260924 10:41:21.738549 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:41:21.738878 2080 external_mini_cluster.cc:1428] Setting key 315429d3617e554b7811758f08c7e64b
Sep 24 10:41:21 dist-test-slave-j38h krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
I20260924 10:41:21.753703 13516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:21.735155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:44203 (local address 127.2.8.1:34383)
0924 10:41:21.736763 (+ 1608us) negotiation.cc:107] Waiting for socket to connect
0924 10:41:21.736828 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 10:41:21.738041 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:41:21.740706 (+ 2665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:41:21.740716 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:41:21.741086 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:41:21.741776 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:41:21.741797 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:41:21.743224 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:41:21.743228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:41:21.744604 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:41:21.744615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:41:21.745365 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:41:21.745422 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:41:21.745610 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:21.746336 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:41:21.746357 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:41:21.748047 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:41:21.750447 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:41:21.750453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:41:21.750471 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:41:21.750826 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:41:21.751139 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:41:21.751143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:41:21.751145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:41:21.751314 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:41:21.751772 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:41:21.751779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:41:21.752099 (+ 320us) client_negotiation.cc:769] Sending connection context
0924 10:41:21.752286 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 10:41:21.752627 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1344,"thread_start_us":139,"threads_started":1}
I20260924 10:41:21.754774 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:21.734683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:34383 (local address 127.2.8.62:44203)
0924 10:41:21.739765 (+ 5082us) server_negotiation.cc:206] Beginning negotiation
0924 10:41:21.739770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:41:21.739785 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:41:21.739825 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:41:21.739828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:41:21.739875 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:41:21.741928 (+ 2053us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:41:21.742041 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:21.742886 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:21.744843 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:21.745234 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:21.748227 (+ 2993us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:21.748367 (+ 140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:41:21.748370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:41:21.748373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:41:21.748391 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:41:21.750271 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:41:21.750952 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:41:21.750955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:41:21.750957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:41:21.751044 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:41:21.751425 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:41:21.751427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:41:21.751429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:41:21.751640 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:41:21.754378 (+ 2738us) server_negotiation.cc:1035] Waiting for connection context
0924 10:41:21.754453 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 10:41:21.754595 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4996,"thread_start_us":92,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
I20260924 10:41:21.756033 13512 heartbeater.cc:347] Connected to a master server at 127.2.8.62:44203
I20260924 10:41:21.756263 13512 heartbeater.cc:464] Registering TS with master...
I20260924 10:41:21.756840 13512 heartbeater.cc:511] Master 127.2.8.62:44203 requested a full tablet report, sending...
I20260924 10:41:21.758585 13300 ts_manager.cc:194] Registered new tserver with Master: 4f2ee7e1ab354f6db28fef54743e0a88 (127.2.8.1:33513)
I20260924 10:41:21.760047 13300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:34383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:41:21.773854 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.8.20:55927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:44203
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:41:21 dist-test-slave-j38h krb5kdc[12824](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790246481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 10:41:21.933977 13520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:41:21.934821 13520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:41:21.934957 13520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:41:21.941119 13520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:41:21.941278 13520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:41:21.941403 13520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:41:21.949396 13520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.8.20:55927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:41:21.951092 13520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:41:21.952440 13520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:41:21.965047 13527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:41:21.965634 13529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:41:21.966662 13520 server_base.cc:1034] running on GCE node
W20260924 10:41:21.969130 13526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:41:21.969676 13520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:41:21.970463 13520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:41:21.971637 13520 hybrid_clock.cc:648] HybridClock initialized: now 1790246481971573 us; error 80 us; skew 500 ppm
Sep 24 10:41:21 dist-test-slave-j38h krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:41:21.975237 13520 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:41:21.976307 13520 webserver.cc:460] Webserver started at http://127.2.8.2:46469/ using document root <none> and password file <none>
I20260924 10:41:21.976959 13520 fs_manager.cc:359] Metadata directory not provided
I20260924 10:41:21.977187 13520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:41:21.977449 13520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:41:21 dist-test-slave-j38h krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:41:21.999382 13520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "2a527a49e0e54a5aa839b86e4939f044"
format_stamp: "Formatted at 2026-09-24 10:41:21 on dist-test-slave-j38h"
server_key: "b4bf88196d36f394537d009e275afcbf"
server_key_iv: "fcedfb5a3f856ae58398818be385b46c"
server_key_version: "kuduclusterkey@0"
I20260924 10:41:22.001174 13520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "2a527a49e0e54a5aa839b86e4939f044"
format_stamp: "Formatted at 2026-09-24 10:41:21 on dist-test-slave-j38h"
server_key: "b4bf88196d36f394537d009e275afcbf"
server_key_iv: "fcedfb5a3f856ae58398818be385b46c"
server_key_version: "kuduclusterkey@0"
I20260924 10:41:22.006301 13520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260924 10:41:22.030807 13536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:41:22.032693 13520 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.000s
I20260924 10:41:22.032831 13520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "2a527a49e0e54a5aa839b86e4939f044"
format_stamp: "Formatted at 2026-09-24 10:41:21 on dist-test-slave-j38h"
server_key: "b4bf88196d36f394537d009e275afcbf"
server_key_iv: "fcedfb5a3f856ae58398818be385b46c"
server_key_version: "kuduclusterkey@0"
I20260924 10:41:22.033001 13520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:41:22.049365 13520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:41:22.053063 13520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:41:22.053325 13520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:41:22.054086 13520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:41:22.055200 13520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:41:22.055306 13520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:41:22.055402 13520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:41:22.055469 13520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:41:22.069336 13520 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:45455
I20260924 10:41:22.070606 13520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:41:22.072263 13649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:45455 every 8 connection(s)
I20260924 10:41:22.073589 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 13520
I20260924 10:41:22.073733 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:41:22.074112 2080 external_mini_cluster.cc:1428] Setting key 9e95a233471cd9be79572ab40d70d695
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
Sep 24 10:41:22 dist-test-slave-j38h krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:41:22.111855 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.076831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:59589 (local address 127.2.8.62:44203)
0924 10:41:22.077059 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 10:41:22.077085 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 10:41:22.085655 (+ 8570us) server_negotiation.cc:407] Connection header received
0924 10:41:22.086947 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:41:22.086951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:41:22.087007 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:41:22.087067 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:41:22.096574 (+ 9507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.098268 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.100949 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.101321 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.101404 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:22.107466 (+ 6062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:41:22.107470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:41:22.107473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:41:22.107505 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:41:22.109378 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:41:22.110051 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:41:22.110055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:41:22.110057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:41:22.110136 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:41:22.110516 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:41:22.110519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:41:22.110521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:41:22.110747 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:41:22.110815 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:41:22.111508 (+ 693us) server_negotiation.cc:299] Negotiation successful
0924 10:41:22.111651 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 10:41:22.113762 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.077592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:44203 (local address 127.2.8.2:59589)
0924 10:41:22.085444 (+ 7852us) negotiation.cc:107] Waiting for socket to connect
0924 10:41:22.085518 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 10:41:22.086678 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:41:22.087499 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:41:22.087509 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:41:22.087884 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:41:22.088551 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:41:22.088569 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.098562 (+ 9993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.098566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:41:22.100752 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:41:22.100766 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.101594 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.101660 (+ 66us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:41:22.101864 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:22.102596 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:41:22.102619 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:41:22.107244 (+ 4625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:41:22.109526 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:41:22.109547 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:41:22.109567 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:41:22.109911 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:41:22.110232 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:41:22.110235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:41:22.110237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:41:22.110407 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:41:22.110889 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:41:22.110896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:41:22.111237 (+ 341us) client_negotiation.cc:769] Sending connection context
0924 10:41:22.112326 (+ 1089us) client_negotiation.cc:240] Negotiation successful
0924 10:41:22.112696 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7567,"thread_start_us":141,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
I20260924 10:41:22.117365 13650 heartbeater.cc:347] Connected to a master server at 127.2.8.62:44203
I20260924 10:41:22.117619 13650 heartbeater.cc:464] Registering TS with master...
I20260924 10:41:22.118163 13650 heartbeater.cc:511] Master 127.2.8.62:44203 requested a full tablet report, sending...
I20260924 10:41:22.119298 13300 ts_manager.cc:194] Registered new tserver with Master: 2a527a49e0e54a5aa839b86e4939f044 (127.2.8.2:45455)
I20260924 10:41:22.120029 13300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:59589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:41:22.141103 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.8.20:55927/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:44203
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:41:22.296022 13655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:41:22.296492 13655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:41:22.296628 13655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:41:22.302956 13655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:41:22.303040 13655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:41:22.303130 13655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:41:22.311074 13655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.8.20:55927/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:44203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:41:22.312458 13655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:41:22.313848 13655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:41:22.325824 13661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:41:22.326309 13655 server_base.cc:1034] running on GCE node
W20260924 10:41:22.325896 13663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:41:22.326646 13660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:41:22.327167 13655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:41:22.327932 13655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:41:22.329113 13655 hybrid_clock.cc:648] HybridClock initialized: now 1790246482329093 us; error 35 us; skew 500 ppm
Sep 24 10:41:22 dist-test-slave-j38h krb5kdc[12824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246482, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:41:22.335078 13655 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:41:22.336107 13655 webserver.cc:460] Webserver started at http://127.2.8.3:33361/ using document root <none> and password file <none>
I20260924 10:41:22.336793 13655 fs_manager.cc:359] Metadata directory not provided
I20260924 10:41:22.336892 13655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:41:22.337150 13655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:41:22 dist-test-slave-j38h krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246482, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:41:22.358563 13655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "65c2da37bcbb4d2db4d4d66ac5f7a415"
format_stamp: "Formatted at 2026-09-24 10:41:22 on dist-test-slave-j38h"
server_key: "4d6a41c6e0e95aa263e5655c3145289d"
server_key_iv: "25ee1e116ea6ad72119518d60cf29851"
server_key_version: "kuduclusterkey@0"
I20260924 10:41:22.359315 13655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "65c2da37bcbb4d2db4d4d66ac5f7a415"
format_stamp: "Formatted at 2026-09-24 10:41:22 on dist-test-slave-j38h"
server_key: "4d6a41c6e0e95aa263e5655c3145289d"
server_key_iv: "25ee1e116ea6ad72119518d60cf29851"
server_key_version: "kuduclusterkey@0"
I20260924 10:41:22.379175 13655 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.006s sys 0.000s
Sep 24 10:41:22 dist-test-slave-j38h krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:41:22.410073 13670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:41:22.412696 13655 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.002s
I20260924 10:41:22.412839 13655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "65c2da37bcbb4d2db4d4d66ac5f7a415"
format_stamp: "Formatted at 2026-09-24 10:41:22 on dist-test-slave-j38h"
server_key: "4d6a41c6e0e95aa263e5655c3145289d"
server_key_iv: "25ee1e116ea6ad72119518d60cf29851"
server_key_version: "kuduclusterkey@0"
I20260924 10:41:22.413009 13655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:41:22.443902 13655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:41:22.447726 13655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:41:22.447985 13655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:41:22.448712 13655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:41:22.449828 13655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:41:22.449937 13655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:41:22.450030 13655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:41:22.450093 13655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:41:22.463966 13655 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:39673
I20260924 10:41:22.465174 13655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:41:22.465973 13783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:39673 every 8 connection(s)
I20260924 10:41:22.472607 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 13655
I20260924 10:41:22.472733 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:41:22.473067 2080 external_mini_cluster.cc:1428] Setting key 67406beccac3708849cf4f761b6f02b7
Sep 24 10:41:22 dist-test-slave-j38h krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246482, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:41:22.484086 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.469178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:43179 (local address 127.2.8.62:44203)
0924 10:41:22.469308 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:41:22.469313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:41:22.470683 (+ 1370us) server_negotiation.cc:407] Connection header received
0924 10:41:22.472200 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:41:22.472204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:41:22.472266 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:41:22.472322 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:41:22.474016 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.474780 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.476439 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.476763 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.476832 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:22.479801 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:41:22.479805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:41:22.479808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:41:22.479836 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:41:22.481657 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:41:22.482296 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:41:22.482300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:41:22.482302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:41:22.482385 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:41:22.482791 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:41:22.482795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:41:22.482796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:41:22.483011 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:41:22.483091 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 10:41:22.483796 (+ 705us) server_negotiation.cc:299] Negotiation successful
0924 10:41:22.483922 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:41:22.485992 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.468819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:44203 (local address 127.2.8.3:43179)
0924 10:41:22.470503 (+ 1684us) negotiation.cc:107] Waiting for socket to connect
0924 10:41:22.470560 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:41:22.471701 (+ 1141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:41:22.472769 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:41:22.472779 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:41:22.473171 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:41:22.473859 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:41:22.473881 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.474933 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.474936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:41:22.476306 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:41:22.476317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.477016 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.477069 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:41:22.477248 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:22.477951 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:41:22.477971 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:41:22.479616 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:41:22.481799 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:41:22.481804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:41:22.481821 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:41:22.482179 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:41:22.482489 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:41:22.482492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:41:22.482494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:41:22.482680 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:41:22.483160 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:41:22.483166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:41:22.483488 (+ 322us) client_negotiation.cc:769] Sending connection context
0924 10:41:22.484573 (+ 1085us) client_negotiation.cc:240] Negotiation successful
0924 10:41:22.484922 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1433,"thread_start_us":503,"threads_started":1}
I20260924 10:41:22.487449 13784 heartbeater.cc:347] Connected to a master server at 127.2.8.62:44203
I20260924 10:41:22.487691 13784 heartbeater.cc:464] Registering TS with master...
I20260924 10:41:22.488232 13784 heartbeater.cc:511] Master 127.2.8.62:44203 requested a full tablet report, sending...
I20260924 10:41:22.489296 13300 ts_manager.cc:194] Registered new tserver with Master: 65c2da37bcbb4d2db4d4d66ac5f7a415 (127.2.8.3:39673)
I20260924 10:41:22.489984 13300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:43179
I20260924 10:41:22.498775 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:41:22.516850 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.505479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.2.8.62:44203)
0924 10:41:22.505613 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:41:22.505619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:41:22.506751 (+ 1132us) server_negotiation.cc:407] Connection header received
0924 10:41:22.506990 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:41:22.506994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:41:22.507047 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:41:22.507105 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:41:22.508199 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.509085 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.510379 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.510726 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.510806 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:22.512848 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:41:22.512852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:41:22.512855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:41:22.512883 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:41:22.514744 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:41:22.515324 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:41:22.515328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:41:22.515329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:41:22.515434 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:41:22.515776 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:41:22.515780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:41:22.515782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:41:22.516025 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:41:22.516139 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 10:41:22.516542 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 10:41:22.516667 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:41:22.521341 13300 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:41:22.524622 13300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:41:22.542551 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.535399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40164 (local address 127.2.8.1:33513)
0924 10:41:22.536403 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0924 10:41:22.536409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:41:22.537193 (+ 784us) server_negotiation.cc:407] Connection header received
0924 10:41:22.537425 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:41:22.537437 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:41:22.537635 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:41:22.537830 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:41:22.538398 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.539416 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.540791 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.541671 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.541781 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:22.541937 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 10:41:22.542296 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 10:41:22.542392 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":102,"threads_started":1}
I20260924 10:41:22.542541 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.534141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:33513 (local address 127.0.0.1:40164)
0924 10:41:22.537059 (+ 2918us) negotiation.cc:107] Waiting for socket to connect
0924 10:41:22.537099 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:41:22.537252 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:41:22.538031 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:41:22.538037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:41:22.538064 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:41:22.538242 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:41:22.538251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.539582 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.539585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:41:22.540653 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:41:22.540663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.542002 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.542004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:41:22.542086 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:22.542115 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:41:22.542258 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 10:41:22.542379 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2701,"thread_start_us":119,"threads_started":1}
I20260924 10:41:22.546003 13445 tablet_service.cc:1467] Processing CreateTablet for tablet e4937e651afd46e999fc1af1c717c532 (DEFAULT_TABLE table=test-table [id=26bce14f338a4eab9aacbac025882e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:41:22.547201 13445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4937e651afd46e999fc1af1c717c532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:41:22.548835 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.536144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:45455 (local address 127.0.0.1:60154)
0924 10:41:22.542903 (+ 6759us) negotiation.cc:107] Waiting for socket to connect
0924 10:41:22.542916 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:41:22.543732 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:41:22.544437 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:41:22.544440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:41:22.544456 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:41:22.544535 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:41:22.544542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.545979 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.545983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:41:22.547045 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:41:22.547058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.548404 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.548408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:41:22.548495 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:22.548521 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 10:41:22.548600 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 10:41:22.548670 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6707,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260924 10:41:22.549398 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.536529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60154 (local address 127.2.8.2:45455)
0924 10:41:22.536813 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0924 10:41:22.536829 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:41:22.543675 (+ 6846us) server_negotiation.cc:407] Connection header received
0924 10:41:22.543902 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:41:22.543907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:41:22.544126 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:41:22.544279 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:41:22.544676 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.545817 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.547225 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.548063 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.548181 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:22.548341 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 10:41:22.549112 (+ 771us) server_negotiation.cc:299] Negotiation successful
0924 10:41:22.549242 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":125,"threads_started":1}
I20260924 10:41:22.549708 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.535020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50622 (local address 127.2.8.3:39673)
0924 10:41:22.536217 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0924 10:41:22.536233 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:41:22.539367 (+ 3134us) server_negotiation.cc:407] Connection header received
0924 10:41:22.539485 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:41:22.539492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:41:22.539710 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:41:22.539871 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:41:22.545114 (+ 5243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.546283 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.547973 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.548832 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.548909 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:22.549049 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 10:41:22.549424 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 10:41:22.549515 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1076,"thread_start_us":115,"threads_started":1}
I20260924 10:41:22.550395 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.534285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:39673 (local address 127.0.0.1:50622)
0924 10:41:22.539246 (+ 4961us) negotiation.cc:107] Waiting for socket to connect
0924 10:41:22.539260 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:41:22.539365 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:41:22.544806 (+ 5441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:41:22.544810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:41:22.544857 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:41:22.544951 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:41:22.544957 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.546430 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.546433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:41:22.547690 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:41:22.547701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.549152 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.549173 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:41:22.549271 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:22.549290 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 10:41:22.550141 (+ 851us) client_negotiation.cc:240] Negotiation successful
0924 10:41:22.550235 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4898,"thread_start_us":79,"threads_started":1}
I20260924 10:41:22.552516 13584 tablet_service.cc:1467] Processing CreateTablet for tablet e4937e651afd46e999fc1af1c717c532 (DEFAULT_TABLE table=test-table [id=26bce14f338a4eab9aacbac025882e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:41:22.553664 13584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4937e651afd46e999fc1af1c717c532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:41:22.553539 13717 tablet_service.cc:1467] Processing CreateTablet for tablet e4937e651afd46e999fc1af1c717c532 (DEFAULT_TABLE table=test-table [id=26bce14f338a4eab9aacbac025882e13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:41:22.554497 13717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4937e651afd46e999fc1af1c717c532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:41:22.561039 13803 tablet_bootstrap.cc:492] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044: Bootstrap starting.
I20260924 10:41:22.562214 13804 tablet_bootstrap.cc:492] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415: Bootstrap starting.
I20260924 10:41:22.564034 13803 tablet_bootstrap.cc:654] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044: Neither blocks nor log segments found. Creating new log.
I20260924 10:41:22.564955 13803 log.cc:824] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044: Log is configured to *not* fsync() on all Append() calls
I20260924 10:41:22.565446 13802 tablet_bootstrap.cc:492] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88: Bootstrap starting.
I20260924 10:41:22.567394 13804 tablet_bootstrap.cc:654] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415: Neither blocks nor log segments found. Creating new log.
I20260924 10:41:22.568742 13804 log.cc:824] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415: Log is configured to *not* fsync() on all Append() calls
I20260924 10:41:22.569855 13802 tablet_bootstrap.cc:654] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88: Neither blocks nor log segments found. Creating new log.
I20260924 10:41:22.570714 13802 log.cc:824] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88: Log is configured to *not* fsync() on all Append() calls
I20260924 10:41:22.573383 13804 tablet_bootstrap.cc:492] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415: No bootstrap required, opened a new log
I20260924 10:41:22.574159 13804 ts_tablet_manager.cc:1397] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260924 10:41:22.578523 13804 raft_consensus.cc:348] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2ee7e1ab354f6db28fef54743e0a88" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33513 } } peers { permanent_uuid: "65c2da37bcbb4d2db4d4d66ac5f7a415" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39673 } } peers { permanent_uuid: "2a527a49e0e54a5aa839b86e4939f044" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45455 } }
I20260924 10:41:22.578927 13804 raft_consensus.cc:374] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:41:22.579036 13804 raft_consensus.cc:729] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c2da37bcbb4d2db4d4d66ac5f7a415, State: Initialized, Role: FOLLOWER
I20260924 10:41:22.579732 13804 consensus_queue.cc:260] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2ee7e1ab354f6db28fef54743e0a88" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33513 } } peers { permanent_uuid: "65c2da37bcbb4d2db4d4d66ac5f7a415" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39673 } } peers { permanent_uuid: "2a527a49e0e54a5aa839b86e4939f044" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45455 } }
I20260924 10:41:22.581396 13804 ts_tablet_manager.cc:1428] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 10:41:22.582955 13802 tablet_bootstrap.cc:492] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88: No bootstrap required, opened a new log
I20260924 10:41:22.583230 13802 ts_tablet_manager.cc:1397] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260924 10:41:22.588049 13802 raft_consensus.cc:348] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2ee7e1ab354f6db28fef54743e0a88" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33513 } } peers { permanent_uuid: "65c2da37bcbb4d2db4d4d66ac5f7a415" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39673 } } peers { permanent_uuid: "2a527a49e0e54a5aa839b86e4939f044" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45455 } }
I20260924 10:41:22.589304 13784 heartbeater.cc:503] Master 127.2.8.62:44203 was elected leader, sending a full tablet report...
I20260924 10:41:22.590308 13803 tablet_bootstrap.cc:492] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044: No bootstrap required, opened a new log
I20260924 10:41:22.590543 13803 ts_tablet_manager.cc:1397] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.002s
I20260924 10:41:22.593072 13802 raft_consensus.cc:374] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:41:22.593142 13802 raft_consensus.cc:729] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f2ee7e1ab354f6db28fef54743e0a88, State: Initialized, Role: FOLLOWER
I20260924 10:41:22.593659 13802 consensus_queue.cc:260] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2ee7e1ab354f6db28fef54743e0a88" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33513 } } peers { permanent_uuid: "65c2da37bcbb4d2db4d4d66ac5f7a415" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39673 } } peers { permanent_uuid: "2a527a49e0e54a5aa839b86e4939f044" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45455 } }
I20260924 10:41:22.594732 13802 ts_tablet_manager.cc:1428] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260924 10:41:22.594666 13803 raft_consensus.cc:348] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2ee7e1ab354f6db28fef54743e0a88" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33513 } } peers { permanent_uuid: "65c2da37bcbb4d2db4d4d66ac5f7a415" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39673 } } peers { permanent_uuid: "2a527a49e0e54a5aa839b86e4939f044" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45455 } }
I20260924 10:41:22.594949 13803 raft_consensus.cc:374] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:41:22.595008 13803 raft_consensus.cc:729] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a527a49e0e54a5aa839b86e4939f044, State: Initialized, Role: FOLLOWER
I20260924 10:41:22.595145 13512 heartbeater.cc:503] Master 127.2.8.62:44203 was elected leader, sending a full tablet report...
I20260924 10:41:22.595628 13803 consensus_queue.cc:260] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2ee7e1ab354f6db28fef54743e0a88" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33513 } } peers { permanent_uuid: "65c2da37bcbb4d2db4d4d66ac5f7a415" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39673 } } peers { permanent_uuid: "2a527a49e0e54a5aa839b86e4939f044" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45455 } }
I20260924 10:41:22.597729 13803 ts_tablet_manager.cc:1428] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260924 10:41:22.598102 13650 heartbeater.cc:503] Master 127.2.8.62:44203 was elected leader, sending a full tablet report...
W20260924 10:41:22.716915 13785 tablet.cc:2369] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:41:22.737192 13513 tablet.cc:2369] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 10:41:22 dist-test-slave-j38h krb5kdc[12824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246481, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:41:22.755375 13809 raft_consensus.cc:482] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:41:22.755600 13809 raft_consensus.cc:504] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2ee7e1ab354f6db28fef54743e0a88" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33513 } } peers { permanent_uuid: "65c2da37bcbb4d2db4d4d66ac5f7a415" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39673 } } peers { permanent_uuid: "2a527a49e0e54a5aa839b86e4939f044" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45455 } }
I20260924 10:41:22.756960 13809 leader_election.cc:288] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65c2da37bcbb4d2db4d4d66ac5f7a415 (127.2.8.3:39673), 2a527a49e0e54a5aa839b86e4939f044 (127.2.8.2:45455)
I20260924 10:41:22.765348 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.758445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:37921 (local address 127.2.8.2:45455)
0924 10:41:22.758578 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:41:22.758583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:41:22.761270 (+ 2687us) server_negotiation.cc:407] Connection header received
0924 10:41:22.761470 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:41:22.761474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:41:22.761519 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:41:22.761578 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:41:22.761942 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.762654 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.764048 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.764631 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.764722 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:22.764784 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 10:41:22.765143 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 10:41:22.765218 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:41:22.765748 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.758121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:45455 (local address 127.2.8.1:37921)
0924 10:41:22.761186 (+ 3065us) negotiation.cc:107] Waiting for socket to connect
0924 10:41:22.761201 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:41:22.761349 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:41:22.761712 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:41:22.761715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:41:22.761730 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:41:22.761826 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:41:22.761831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.762791 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.762794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:41:22.763911 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:41:22.763921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.764874 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.764877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:41:22.764968 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:22.765017 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 10:41:22.765503 (+ 486us) client_negotiation.cc:240] Negotiation successful
0924 10:41:22.765591 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2991,"thread_start_us":117,"threads_started":1}
I20260924 10:41:22.766603 13603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4937e651afd46e999fc1af1c717c532" candidate_uuid: "4f2ee7e1ab354f6db28fef54743e0a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a527a49e0e54a5aa839b86e4939f044" is_pre_election: true
I20260924 10:41:22.766939 13603 raft_consensus.cc:2454] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f2ee7e1ab354f6db28fef54743e0a88 in term 0.
I20260924 10:41:22.767284 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.757876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:38469 (local address 127.2.8.3:39673)
0924 10:41:22.758050 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 10:41:22.758076 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 10:41:22.758541 (+ 465us) server_negotiation.cc:407] Connection header received
0924 10:41:22.758753 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:41:22.758757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:41:22.758807 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:41:22.758854 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:41:22.759868 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.760669 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.763413 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.764051 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.764122 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:22.764187 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 10:41:22.767074 (+ 2887us) server_negotiation.cc:299] Negotiation successful
0924 10:41:22.767162 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:41:22.767863 13402 leader_election.cc:302] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a527a49e0e54a5aa839b86e4939f044, 4f2ee7e1ab354f6db28fef54743e0a88; no voters:
I20260924 10:41:22.768018 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.757320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:39673 (local address 127.2.8.1:38469)
0924 10:41:22.758449 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0924 10:41:22.758465 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:41:22.758627 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:41:22.759120 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:41:22.759123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:41:22.759144 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:41:22.759545 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:41:22.759554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.761225 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.761231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:41:22.762514 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:41:22.762523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.764273 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.764276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:41:22.764611 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:22.766928 (+ 2317us) client_negotiation.cc:769] Sending connection context
0924 10:41:22.767746 (+ 818us) client_negotiation.cc:240] Negotiation successful
0924 10:41:22.767826 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"thread_start_us":115,"threads_started":1}
I20260924 10:41:22.768306 13809 raft_consensus.cc:2781] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:41:22.768386 13809 raft_consensus.cc:482] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:41:22.768427 13809 raft_consensus.cc:3037] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:41:22.769425 13722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4937e651afd46e999fc1af1c717c532" candidate_uuid: "4f2ee7e1ab354f6db28fef54743e0a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c2da37bcbb4d2db4d4d66ac5f7a415" is_pre_election: true
I20260924 10:41:22.769711 13722 raft_consensus.cc:2454] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f2ee7e1ab354f6db28fef54743e0a88 in term 0.
I20260924 10:41:22.771199 13809 raft_consensus.cc:504] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2ee7e1ab354f6db28fef54743e0a88" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33513 } } peers { permanent_uuid: "65c2da37bcbb4d2db4d4d66ac5f7a415" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39673 } } peers { permanent_uuid: "2a527a49e0e54a5aa839b86e4939f044" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45455 } }
I20260924 10:41:22.771607 13809 leader_election.cc:288] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [CANDIDATE]: Term 1 election: Requested vote from peers 65c2da37bcbb4d2db4d4d66ac5f7a415 (127.2.8.3:39673), 2a527a49e0e54a5aa839b86e4939f044 (127.2.8.2:45455)
I20260924 10:41:22.772468 13722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4937e651afd46e999fc1af1c717c532" candidate_uuid: "4f2ee7e1ab354f6db28fef54743e0a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65c2da37bcbb4d2db4d4d66ac5f7a415"
I20260924 10:41:22.772595 13722 raft_consensus.cc:3037] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:41:22.773958 13722 raft_consensus.cc:2454] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f2ee7e1ab354f6db28fef54743e0a88 in term 1.
I20260924 10:41:22.774297 13399 leader_election.cc:302] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f2ee7e1ab354f6db28fef54743e0a88, 65c2da37bcbb4d2db4d4d66ac5f7a415; no voters:
I20260924 10:41:22.774942 13603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4937e651afd46e999fc1af1c717c532" candidate_uuid: "4f2ee7e1ab354f6db28fef54743e0a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a527a49e0e54a5aa839b86e4939f044"
I20260924 10:41:22.775074 13603 raft_consensus.cc:3037] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:41:22.776826 13603 raft_consensus.cc:2454] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f2ee7e1ab354f6db28fef54743e0a88 in term 1.
I20260924 10:41:22.777354 13809 raft_consensus.cc:2781] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:41:22.777631 13809 raft_consensus.cc:686] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [term 1 LEADER]: Becoming Leader. State: Replica: 4f2ee7e1ab354f6db28fef54743e0a88, State: Running, Role: LEADER
I20260924 10:41:22.778021 13809 consensus_queue.cc:237] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2ee7e1ab354f6db28fef54743e0a88" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33513 } } peers { permanent_uuid: "65c2da37bcbb4d2db4d4d66ac5f7a415" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39673 } } peers { permanent_uuid: "2a527a49e0e54a5aa839b86e4939f044" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45455 } }
I20260924 10:41:22.781661 13298 catalog_manager.cc:5495] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f2ee7e1ab354f6db28fef54743e0a88 (127.2.8.1). New cstate: current_term: 1 leader_uuid: "4f2ee7e1ab354f6db28fef54743e0a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f2ee7e1ab354f6db28fef54743e0a88" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65c2da37bcbb4d2db4d4d66ac5f7a415" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a527a49e0e54a5aa839b86e4939f044" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45455 } health_report { overall_health: UNKNOWN } } }
W20260924 10:41:22.829550 13651 tablet.cc:2369] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:41:22.864576 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.849060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.2.8.1:33513)
0924 10:41:22.849199 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 10:41:22.849210 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:41:22.849304 (+ 94us) server_negotiation.cc:407] Connection header received
0924 10:41:22.849493 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:41:22.849497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:41:22.849566 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:41:22.849684 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:41:22.850049 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.850864 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.862220 (+ 11356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:41:22.862588 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:41:22.862712 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:41:22.863237 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:41:22.863856 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:41:22.863983 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 10:41:22.864359 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 10:41:22.864425 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:41:22.874294 13722 raft_consensus.cc:1261] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415 [term 1 FOLLOWER]: Refusing update from remote peer 4f2ee7e1ab354f6db28fef54743e0a88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:41:22.875538 13603 raft_consensus.cc:1261] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044 [term 1 FOLLOWER]: Refusing update from remote peer 4f2ee7e1ab354f6db28fef54743e0a88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:41:22.875635 13814 consensus_queue.cc:1035] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65c2da37bcbb4d2db4d4d66ac5f7a415" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:41:22.876332 13814 consensus_queue.cc:1035] T e4937e651afd46e999fc1af1c717c532 P 4f2ee7e1ab354f6db28fef54743e0a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a527a49e0e54a5aa839b86e4939f044" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 45455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:41:22.889882 13818 mvcc.cc:205] Tried to move back new op lower bound from 7332849593839734784 to 7332849593462484992. Current Snapshot: MvccSnapshot[applied={T|T < 7332849593839734784}]
I20260924 10:41:22.893541 13820 mvcc.cc:205] Tried to move back new op lower bound from 7332849593839734784 to 7332849593462484992. Current Snapshot: MvccSnapshot[applied={T|T < 7332849593839734784}]
I20260924 10:41:22.899778 13819 mvcc.cc:205] Tried to move back new op lower bound from 7332849593839734784 to 7332849593462484992. Current Snapshot: MvccSnapshot[applied={T|T < 7332849593839734784}]
I20260924 10:41:22.901265 13298 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:41:22.901448 13298 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:41:22.904247 13298 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8fde9a21bf6d40428deea067c37315a3: Sending DeleteTablet for 3 replicas of tablet e4937e651afd46e999fc1af1c717c532
I20260924 10:41:22.905460 13584 tablet_service.cc:1514] Processing DeleteTablet for tablet e4937e651afd46e999fc1af1c717c532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:41:22 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:60154
I20260924 10:41:22.906056 13445 tablet_service.cc:1514] Processing DeleteTablet for tablet e4937e651afd46e999fc1af1c717c532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:41:22 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:40164
I20260924 10:41:22.906812 13717 tablet_service.cc:1514] Processing DeleteTablet for tablet e4937e651afd46e999fc1af1c717c532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:41:22 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:50622
I20260924 10:41:22.908046 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 13381
I20260924 10:41:22.909179 13827 tablet_replica.cc:330] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044: stopping tablet replica
I20260924 10:41:22.909461 13829 tablet_replica.cc:330] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415: stopping tablet replica
I20260924 10:41:22.909507 13827 raft_consensus.cc:2229] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:41:22.909720 13829 raft_consensus.cc:2229] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:41:22.909839 13827 raft_consensus.cc:2258] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:41:22.909948 13829 raft_consensus.cc:2258] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:41:22.912918 13829 ts_tablet_manager.cc:1905] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:41:22.918788 13829 ts_tablet_manager.cc:1918] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:41:22.919070 13829 log.cc:1196] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/e4937e651afd46e999fc1af1c717c532
I20260924 10:41:22.919409 13829 ts_tablet_manager.cc:1939] T e4937e651afd46e999fc1af1c717c532 P 65c2da37bcbb4d2db4d4d66ac5f7a415: Deleting consensus metadata
I20260924 10:41:22.920106 13827 ts_tablet_manager.cc:1905] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 10:41:22.921994 13285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.8.1:33513 (error 108)
I20260924 10:41:22.922706 13284 catalog_manager.cc:4841] TS 65c2da37bcbb4d2db4d4d66ac5f7a415 (127.2.8.3:39673): tablet e4937e651afd46e999fc1af1c717c532 (table test-table [id=26bce14f338a4eab9aacbac025882e13]) successfully deleted
I20260924 10:41:22.922874 13827 ts_tablet_manager.cc:1918] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:41:22.922973 13827 log.cc:1196] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/e4937e651afd46e999fc1af1c717c532
I20260924 10:41:22.923333 13827 ts_tablet_manager.cc:1939] T e4937e651afd46e999fc1af1c717c532 P 2a527a49e0e54a5aa839b86e4939f044: Deleting consensus metadata
I20260924 10:41:22.923569 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 13520
I20260924 10:41:22.925295 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:41:22.924914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:33513 (local address 127.0.0.1:40170)
0924 10:41:22.925088 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 10:41:22.925207 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:33513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260924 10:41:22.925577 13285 catalog_manager.cc:4584] TS 4f2ee7e1ab354f6db28fef54743e0a88 (127.2.8.1:33513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4937e651afd46e999fc1af1c717c532: Network error: Client connection negotiation failed: client connection to 127.2.8.1:33513: connect: Connection refused (error 111)
W20260924 10:41:22.933389 13800 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:41:22.932715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:45455 (local address 127.0.0.1:60158)
0924 10:41:22.932894 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 10:41:22.932910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:41:22.933037 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:41:22.933270 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.2:45455: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 10:41:22.933979 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 13655
I20260924 10:41:22.942600 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 13268
I20260924 10:41:22.948191 2080 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 10:41:23.293242 2080 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 10:41:23.293375 2080 mini_ranger.cc:66] Stopping Ranger...
I20260924 10:41:23.628964 2080 mini_ranger.cc:68] Stopped Ranger
2026-09-24 10:41:23.629 UTC [12854] LOG: received smart shutdown request
2026-09-24 10:41:23.632 UTC [12854] LOG: background worker "logical replication launcher" (PID 12863) exited with exit code 1
2026-09-24 10:41:23.633 UTC [12858] LOG: shutting down
2026-09-24 10:41:23.818 UTC [12854] LOG: database system is shut down
2026-09-24T10:41:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47616 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:41:24 dist-test-slave-j38h krb5kdc[13839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:41:24 dist-test-slave-j38h krb5kdc[13839](info): set up 2 sockets
Sep 24 10:41:24 dist-test-slave-j38h krb5kdc[13839](info): commencing operation
krb5kdc: starting...
W20260924 10:41:24.985080 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:41:25 dist-test-slave-j38h krb5kdc[13839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:41:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:41:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.8.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.8.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.8.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/rangeradmin_127.2.8.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.8.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/rangeradmin_127.2.8.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.8.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.8.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.8.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/rangerlookup_127.2.8.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.8.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/rangerlookup_127.2.8.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.8.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.8.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.8.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/HTTP_127.2.8.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.8.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/HTTP_127.2.8.20@KRBTEST.COM.keytab.
I20260924 10:41:25.159879 2080 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2SD_Oz/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/postgres -l logfile start
2026-09-24 10:41:27.720 UTC [13869] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 10:41:27.720 UTC [13869] LOG: listening on IPv4 address "127.2.8.20", port 58321
2026-09-24 10:41:27.724 UTC [13869] LOG: listening on Unix socket "/tmp/.s.PGSQL.58321"
2026-09-24 10:41:27.759 UTC [13872] LOG: database system was shut down at 2026-09-24 10:41:26 UTC
2026-09-24 10:41:27.764 UTC [13869] LOG: database system is ready to accept connections
I20260924 10:41:28.720831 2080 mini_postgres.cc:96] Postgres bound to 58321
2026-09-24 10:41:28.727 UTC [13880] FATAL: database "slave" does not exist
127.2.8.20:58321 - accepting connections
I20260924 10:41:28.727849 2080 mini_ranger.cc:161] Starting Ranger...
I20260924 10:41:28.742736 2080 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 10:41:29.252852 2080 mini_ranger.cc:87] Created ranger Postgres database
I20260924 10:41:29.252960 2080 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-admin
2026-09-24 10:41:29,589 [I] DB FLAVOR :POSTGRES
2026-09-24 10:41:29,589 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:41:29,589 [I] Checking connection..
2026-09-24 10:41:29,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 10:41:29,829 [I] Checking connection passed.
2026-09-24 10:41:29,829 [I] --------- Verifying version history table ---------
2026-09-24 10:41:29,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:41:30,062 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:41:30,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:41:30,316 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 10:41:30,316 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 10:41:30,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 10:41:30.571 UTC [13944] WARNING: there is no transaction in progress
2026-09-24 10:41:30,577 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 10:41:30,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 10:41:30,831 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 10:41:30,831 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:41:30,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 10:41:31,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 10:41:31,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j38h.c.gcp-upstream.internal','N') ;"
2026-09-24 10:41:31,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:41:31,826 [I] Table x_portal_user does not exist in database ranger
2026-09-24 10:41:31,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:41:32,067 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 10:41:32,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:41:32,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:41:32,548 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 10:41:32,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 10:41:33.530 UTC [14052] WARNING: there is no transaction in progress
2026-09-24 10:41:33.547 UTC [14052] WARNING: there is no transaction in progress
2026-09-24 10:41:33.563 UTC [14052] WARNING: there is no transaction in progress
2026-09-24 10:41:33.579 UTC [14052] WARNING: there is no transaction in progress
2026-09-24 10:41:33.598 UTC [14052] WARNING: there is no transaction in progress
2026-09-24 10:41:33.618 UTC [14052] WARNING: there is no transaction in progress
2026-09-24 10:41:33.680 UTC [14052] WARNING: there is no transaction in progress
2026-09-24 10:41:33.687 UTC [14052] WARNING: there is no transaction in progress
2026-09-24 10:41:33.701 UTC [14052] WARNING: there is no transaction in progress
2026-09-24 10:41:33.712 UTC [14052] WARNING: there is no transaction in progress
2026-09-24 10:41:33.722 UTC [14052] WARNING: there is no transaction in progress
2026-09-24 10:41:33.736 UTC [14052] WARNING: there is no transaction in progress
2026-09-24 10:41:33.745 UTC [14052] WARNING: there is no transaction in progress
2026-09-24 10:41:33.751 UTC [14052] WARNING: there is no transaction in progress
2026-09-24 10:41:33.758 UTC [14052] WARNING: there is no transaction in progress
2026-09-24 10:41:33.765 UTC [14052] WARNING: there is no transaction in progress
2026-09-24 10:41:34,417 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 10:41:34,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 10:41:34,645 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 10:41:34,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 10:41:34,898 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 10:41:34,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 10:41:35,150 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 10:41:35,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 10:41:35,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 10:41:35,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j38h.c.gcp-upstream.internal';"
2026-09-24 10:41:35,874 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 10:41:35,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 10:41:36,098 [I] DB_PATCHES have already been applied
I20260924 10:41:36.104481 2080 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-admin:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2SD_Oz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 10:41:36.104547 2080 mini_ranger.cc:193] Using host: 127.2.8.20
I20260924 10:41:36.107970 2080 mini_ranger.cc:227] Ranger admin URL: http://127.2.8.20:56823
Sep 24, 2026 10:41:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:41:36 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:41:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 10:41:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 10:41:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 10:41:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 10:41:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 10:41:37.250705 2080 mini_ranger.cc:160] Time spent starting Ranger: real 8.523s user 0.005s sys 0.004s
Sep 24, 2026 10:41:57 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 10:41:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 10:41:57 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 10:41:57 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 10:41:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 10:41:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 10:41:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 10:41:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 10:41:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 10:41:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 10:41:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 10:41:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 10:41:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 10:41:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 10:41:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 10:41:57 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 10:41:57 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 10:41:58 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 10:41:59.060506 2080 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.8.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.8.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.8.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/rangerkms_127.2.8.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.8.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/rangerkms_127.2.8.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.8.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.8.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.8.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/HTTP_127.2.8.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.8.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/HTTP_127.2.8.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 10:41:59 dist-test-slave-j38h krb5kdc[13839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246519, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 10:41:59.141041 2080 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 10:41:59.150611 2080 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 10:42:00.011152 2080 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 10:42:00.011240 2080 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-kms
I20260924 10:42:00.011263 2080 mini_ranger_kms.cc:227] Using postgres at 127.2.8.20:58321
2026-09-24 10:42:00,179 [I] DB FLAVOR :POSTGRES
2026-09-24 10:42:00,179 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 10:42:00,179 [I] Checking connection
2026-09-24 10:42:00,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 10:42:00,418 [I] connection success
2026-09-24 10:42:00,419 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 10:42:00,419 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 10:42:00,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 10:42:00,662 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 10:42:00,662 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 10:42:00,662 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 10:42:00,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.8.20:58321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 10:42:00,932 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 10:42:01.370504 2080 mini_ranger_kms.cc:310] Created kms service
I20260924 10:42:01.590587 2080 mini_ranger_kms.cc:326] Created kudu user
I20260924 10:42:01.634053 2080 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 10:42:01 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246519, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:42:02.036783 2080 mini_ranger_kms.cc:384] Added ranger policy
I20260924 10:42:02.036943 2080 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-kms:/tmp/dist-test-task2SD_Oz/build/debug/bin/postgresql.jar:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2SD_Oz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 10:42:02.036993 2080 mini_ranger_kms.cc:240] Using host: 127.2.8.20
I20260924 10:42:02.040771 2080 mini_ranger_kms.cc:276] Ranger KMS PID: 14232
I20260924 10:42:02.040863 2080 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.8.20:40275
Sep 24, 2026 10:42:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 10:42:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 10:42:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 10:42:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 10:42:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 10:42:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 10:42:04.340654 2080 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.200s user 0.006s sys 0.146s
I20260924 10:42:04.340787 2080 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 10:42:04.340868 2080 mini_ranger_kms.cc:401] 127.2.8.20:40275/kms/v1/keys
Sep 24 10:42:04 dist-test-slave-j38h krb5kdc[13839](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790246524, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.8.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 10:42:04 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246524, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.8.20@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
Sep 24 10:42:05 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246524, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.8.20@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
Sep 24 10:42:07 dist-test-slave-j38h krb5kdc[13839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:42:07.393172 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:38111
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:38111
--ranger_config_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.8.20:40275/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:07.507551 14283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:07.507812 14283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:07.507851 14283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:07.511838 14283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:42:07.511895 14283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:07.511914 14283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:07.511930 14283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:42:07.511945 14283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:42:07.517239 14283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.8.20:40275/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:38111
--ranger_config_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:38111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:07.518292 14283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:07.519315 14283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:07.527506 14288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:07.527490 14291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:07.527817 14283 server_base.cc:1034] running on GCE node
W20260924 10:42:07.528065 14289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:07.528604 14283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:07.529753 14283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:07.530937 14283 hybrid_clock.cc:648] HybridClock initialized: now 1790246527530915 us; error 43 us; skew 500 ppm
Sep 24 10:42:07 dist-test-slave-j38h krb5kdc[13839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246527, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:07.543334 14283 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:42:07.544371 14283 webserver.cc:460] Webserver started at http://127.2.8.62:35547/ using document root <none> and password file <none>
I20260924 10:42:07.544953 14283 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:07.545068 14283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:07.545297 14283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:42:07 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246527, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:42:07.607890 14283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "510c721fd0344cd8b40bc79ec83d1c2d"
format_stamp: "Formatted at 2026-09-24 10:42:07 on dist-test-slave-j38h"
server_key: "6b11b606e235419b53626f31c97b843e"
server_key_iv: "c150127b3d13cd0351a74f82e6c3e586"
server_key_version: "kuduclusterkey@0"
I20260924 10:42:07.608428 14283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "510c721fd0344cd8b40bc79ec83d1c2d"
format_stamp: "Formatted at 2026-09-24 10:42:07 on dist-test-slave-j38h"
server_key: "6b11b606e235419b53626f31c97b843e"
server_key_iv: "c150127b3d13cd0351a74f82e6c3e586"
server_key_version: "kuduclusterkey@0"
I20260924 10:42:07.612776 14283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:42:07.630719 14298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:07.632107 14283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:42:07.632208 14283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "510c721fd0344cd8b40bc79ec83d1c2d"
format_stamp: "Formatted at 2026-09-24 10:42:07 on dist-test-slave-j38h"
server_key: "6b11b606e235419b53626f31c97b843e"
server_key_iv: "c150127b3d13cd0351a74f82e6c3e586"
server_key_version: "kuduclusterkey@0"
I20260924 10:42:07.632362 14283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:07.658056 14283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:07.661093 14283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:07.661291 14283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:07.669273 14283 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:38111
I20260924 10:42:07.669279 14350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:38111 every 8 connection(s)
I20260924 10:42:07.670251 14283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:42:07.671267 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 14283
I20260924 10:42:07.671397 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:42:07.671705 2080 external_mini_cluster.cc:1428] Setting key 413b9c2cc81f6bb17948451be351ae14
I20260924 10:42:07.673470 14351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:07.680004 14351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d: Bootstrap starting.
Sep 24 10:42:07 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:07.682590 14351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:07.683542 14351 log.cc:824] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:07.686208 14351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d: No bootstrap required, opened a new log
I20260924 10:42:07.689069 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:07.673158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.2.8.62:38111)
0924 10:42:07.673622 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:07.673641 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:07.673682 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:42:07.674491 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:07.674513 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:07.674929 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:07.675286 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:07.676189 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:07.677698 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:07.678936 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:07.679620 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:07.679769 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:07.682858 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:07.682866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:07.682886 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:07.682919 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:07.685554 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:07.686128 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:07.686134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:07.686139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:07.686271 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:07.686630 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:07.686633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:07.686634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:07.687122 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:07.687320 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:07.687623 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 10:42:07.688060 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":137,"threads_started":1}
I20260924 10:42:07.690114 14351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510c721fd0344cd8b40bc79ec83d1c2d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38111 } }
I20260924 10:42:07.690400 14351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:07.690475 14351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510c721fd0344cd8b40bc79ec83d1c2d, State: Initialized, Role: FOLLOWER
I20260924 10:42:07.690940 14351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510c721fd0344cd8b40bc79ec83d1c2d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38111 } }
I20260924 10:42:07.691088 14351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:42:07.691190 14351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:42:07.691329 14351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:07.692276 14351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510c721fd0344cd8b40bc79ec83d1c2d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38111 } }
I20260924 10:42:07.692610 14351 leader_election.cc:302] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 510c721fd0344cd8b40bc79ec83d1c2d; no voters:
I20260924 10:42:07.692924 14351 leader_election.cc:288] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:42:07.693058 14356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:42:07.693262 14356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 1 LEADER]: Becoming Leader. State: Replica: 510c721fd0344cd8b40bc79ec83d1c2d, State: Running, Role: LEADER
I20260924 10:42:07.693681 14356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510c721fd0344cd8b40bc79ec83d1c2d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38111 } }
I20260924 10:42:07.694187 14351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:42:07.694902 14356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 510c721fd0344cd8b40bc79ec83d1c2d. Latest consensus state: current_term: 1 leader_uuid: "510c721fd0344cd8b40bc79ec83d1c2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510c721fd0344cd8b40bc79ec83d1c2d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38111 } } }
I20260924 10:42:07.695845 14356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:07.695737 14357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "510c721fd0344cd8b40bc79ec83d1c2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510c721fd0344cd8b40bc79ec83d1c2d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38111 } } }
I20260924 10:42:07.696101 14357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:07.696363 14369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:42:07.696483 14351 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 10:42:07.702592 14369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:42:07.711277 14369 catalog_manager.cc:1347] Generated new cluster ID: 71fc3c056e0043dbb05cee2244aa8980
I20260924 10:42:07.711345 14369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:42:07.728309 14369 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:42:07.729597 14369 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:42:07.736773 14369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d: Generated new TSK 0
I20260924 10:42:07.737617 14369 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:42:08.500737 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.8.20:40275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:38111
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:08.620872 14396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:08.621237 14396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:08.621351 14396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:08.625787 14396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:08.625872 14396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:08.625976 14396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:42:08.631819 14396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.8.20:40275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:38111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:08.633127 14396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:08.634562 14396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:08.645186 14401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:08.645186 14402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:08.645190 14404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:08.646045 14396 server_base.cc:1034] running on GCE node
I20260924 10:42:08.646471 14396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:08.647215 14396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:08.648428 14396 hybrid_clock.cc:648] HybridClock initialized: now 1790246528648374 us; error 73 us; skew 500 ppm
Sep 24 10:42:08 dist-test-slave-j38h krb5kdc[13839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246528, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:08.651866 14396 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:42:08.652941 14396 webserver.cc:460] Webserver started at http://127.2.8.1:46225/ using document root <none> and password file <none>
I20260924 10:42:08.653612 14396 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:08.653718 14396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:08.654000 14396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:42:08 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246528, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:42:08.671499 14396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "f853cfd98be44b7fb1b80153045fcd2e"
format_stamp: "Formatted at 2026-09-24 10:42:08 on dist-test-slave-j38h"
server_key: "b8d3bd3499c186d22a6f4ec0b490b1b2"
server_key_iv: "ad7cb2cad655ea7cc89cfdf31649d647"
server_key_version: "kuduclusterkey@0"
I20260924 10:42:08.672385 14396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "f853cfd98be44b7fb1b80153045fcd2e"
format_stamp: "Formatted at 2026-09-24 10:42:08 on dist-test-slave-j38h"
server_key: "b8d3bd3499c186d22a6f4ec0b490b1b2"
server_key_iv: "ad7cb2cad655ea7cc89cfdf31649d647"
server_key_version: "kuduclusterkey@0"
I20260924 10:42:08.677639 14396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:42:08.704542 14413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:08.706732 14396 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260924 10:42:08.706887 14396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "f853cfd98be44b7fb1b80153045fcd2e"
format_stamp: "Formatted at 2026-09-24 10:42:08 on dist-test-slave-j38h"
server_key: "b8d3bd3499c186d22a6f4ec0b490b1b2"
server_key_iv: "ad7cb2cad655ea7cc89cfdf31649d647"
server_key_version: "kuduclusterkey@0"
I20260924 10:42:08.707116 14396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:08.730078 14396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:08.734269 14396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:08.734481 14396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:08.735301 14396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:08.736447 14396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:42:08.736518 14396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:08.736580 14396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:08.736670 14396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:08.748695 14396 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:35451
I20260924 10:42:08.750000 14396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:42:08.751376 14526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:35451 every 8 connection(s)
Sep 24 10:42:08 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246528, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:08.765283 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 14396
I20260924 10:42:08.765425 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:42:08.765780 2080 external_mini_cluster.cc:1428] Setting key 92f9971eb3ebacf8004564ea9eba9b98
I20260924 10:42:08.765863 14529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:08.752124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:52711 (local address 127.2.8.62:38111)
0924 10:42:08.752416 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:08.752421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:08.753238 (+ 817us) server_negotiation.cc:407] Connection header received
0924 10:42:08.754211 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:08.754214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:08.754270 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:08.754361 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:08.755485 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:08.756093 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:08.757568 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:08.757909 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:08.757990 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:08.761173 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:08.761177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:08.761180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:08.761220 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:08.763316 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:08.763956 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:08.763960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:08.763962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:08.764047 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:08.764474 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:08.764478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:08.764480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:08.764708 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:08.764790 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:08.765543 (+ 753us) server_negotiation.cc:299] Negotiation successful
0924 10:42:08.765686 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":99,"threads_started":1}
I20260924 10:42:08.767493 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:08.752565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38111 (local address 127.2.8.1:52711)
0924 10:42:08.753078 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:08.753129 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:08.754005 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:08.754547 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:08.754554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:08.754865 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:08.755341 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:08.755354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:08.756228 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:08.756232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:08.757426 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:08.757437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:08.758145 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:08.758215 (+ 70us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:08.758413 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:08.759116 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:08.759137 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:08.760929 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:08.763475 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:08.763482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:08.763501 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:08.763838 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:08.764170 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:08.764173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:08.764175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:08.764361 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:08.764855 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:08.764862 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:08.765247 (+ 385us) client_negotiation.cc:769] Sending connection context
0924 10:42:08.766084 (+ 837us) client_negotiation.cc:240] Negotiation successful
0924 10:42:08.766448 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":143,"threads_started":1}
I20260924 10:42:08.770371 14527 heartbeater.cc:347] Connected to a master server at 127.2.8.62:38111
I20260924 10:42:08.770606 14527 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:08.771159 14527 heartbeater.cc:511] Master 127.2.8.62:38111 requested a full tablet report, sending...
I20260924 10:42:08.772694 14315 ts_manager.cc:194] Registered new tserver with Master: f853cfd98be44b7fb1b80153045fcd2e (127.2.8.1:35451)
I20260924 10:42:08.773856 14315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:52711
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:42:08.808185 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.8.20:40275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:38111
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:42:08 dist-test-slave-j38h krb5kdc[13839](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790246528, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 10:42:08.977883 14535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:08.978262 14535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:08.978330 14535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:08.984488 14535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:08.984571 14535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:08.984658 14535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:42:08.991408 14535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.8.20:40275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:38111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:08.992763 14535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:08.994160 14535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:09.006790 14541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:09.006824 14542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:09.007839 14535 server_base.cc:1034] running on GCE node
W20260924 10:42:09.009469 14544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:09.010082 14535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:09.010931 14535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:09.014349 14535 hybrid_clock.cc:648] HybridClock initialized: now 1790246529014264 us; error 83 us; skew 500 ppm
Sep 24 10:42:09 dist-test-slave-j38h krb5kdc[13839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246529, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:09.018170 14535 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:42:09.019320 14535 webserver.cc:460] Webserver started at http://127.2.8.2:42291/ using document root <none> and password file <none>
I20260924 10:42:09.020062 14535 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:09.020211 14535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:09.020509 14535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:42:09 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246529, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:42:09.043645 14535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "95825a87dcba45fe8ac7235fbcb76d16"
format_stamp: "Formatted at 2026-09-24 10:42:09 on dist-test-slave-j38h"
server_key: "cf2e4c5cb063cd3e5ead59d3b27b9abf"
server_key_iv: "3527e8d9fc9b946482e4f808a629b204"
server_key_version: "kuduclusterkey@0"
I20260924 10:42:09.044497 14535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "95825a87dcba45fe8ac7235fbcb76d16"
format_stamp: "Formatted at 2026-09-24 10:42:09 on dist-test-slave-j38h"
server_key: "cf2e4c5cb063cd3e5ead59d3b27b9abf"
server_key_iv: "3527e8d9fc9b946482e4f808a629b204"
server_key_version: "kuduclusterkey@0"
I20260924 10:42:09.049615 14535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 10:42:09.081418 14551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:09.083416 14535 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.002s sys 0.000s
I20260924 10:42:09.083551 14535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "95825a87dcba45fe8ac7235fbcb76d16"
format_stamp: "Formatted at 2026-09-24 10:42:09 on dist-test-slave-j38h"
server_key: "cf2e4c5cb063cd3e5ead59d3b27b9abf"
server_key_iv: "3527e8d9fc9b946482e4f808a629b204"
server_key_version: "kuduclusterkey@0"
I20260924 10:42:09.083704 14535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:09.116158 14535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:09.119937 14535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:09.120150 14535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:09.120891 14535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:09.121922 14535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:42:09.121984 14535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:09.122035 14535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:09.122117 14535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 10:42:09.132748 14535 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:40517
I20260924 10:42:09.132793 14664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:40517 every 8 connection(s)
I20260924 10:42:09.134047 14535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:42:09.138171 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 14535
I20260924 10:42:09.138326 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:42:09.138720 2080 external_mini_cluster.cc:1428] Setting key e50466769a49e714748773f99851b095
Sep 24 10:42:09 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246529, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
I20260924 10:42:09.181468 14529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:09.136532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:48593 (local address 127.2.8.62:38111)
0924 10:42:09.136688 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:09.136695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:09.141475 (+ 4780us) server_negotiation.cc:407] Connection header received
0924 10:42:09.142796 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:09.142799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:09.142851 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:09.142943 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:09.144404 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:09.145274 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:09.167575 (+ 22301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:09.167957 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:09.168083 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:09.173786 (+ 5703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:09.173790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:09.173794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:09.173826 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:09.175661 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:09.179267 (+ 3606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:09.179271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:09.179273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:09.179367 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:09.179851 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:09.179854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:09.179856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:09.180087 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:09.180193 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:09.181128 (+ 935us) server_negotiation.cc:299] Negotiation successful
0924 10:42:09.181270 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
I20260924 10:42:09.184763 14667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:09.137126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38111 (local address 127.2.8.2:48593)
0924 10:42:09.141319 (+ 4193us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:09.141378 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:09.142546 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:09.143165 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:09.143175 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:09.143558 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:09.144236 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:09.144256 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:09.165711 (+ 21455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:09.165717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:09.167372 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:09.167393 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:09.168234 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:09.168304 (+ 70us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:09.168548 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:09.169345 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:09.169369 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:09.173540 (+ 4171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:09.178714 (+ 5174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:09.178721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:09.178742 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:09.179128 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:09.179490 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:09.179493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:09.179495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:09.179729 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:09.180249 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:09.180271 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:09.180627 (+ 356us) client_negotiation.cc:769] Sending connection context
0924 10:42:09.181657 (+ 1030us) client_negotiation.cc:240] Negotiation successful
0924 10:42:09.183641 (+ 1984us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3940,"thread_start_us":139,"threads_started":1}
I20260924 10:42:09.186610 14665 heartbeater.cc:347] Connected to a master server at 127.2.8.62:38111
I20260924 10:42:09.186869 14665 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:09.187481 14665 heartbeater.cc:511] Master 127.2.8.62:38111 requested a full tablet report, sending...
I20260924 10:42:09.188608 14315 ts_manager.cc:194] Registered new tserver with Master: 95825a87dcba45fe8ac7235fbcb76d16 (127.2.8.2:40517)
I20260924 10:42:09.189399 14315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:48593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:42:09.219887 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.8.20:40275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:38111
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:42:09 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246528, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
W20260924 10:42:09.385077 14670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:09.385838 14670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:09.385963 14670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:09.398093 14670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:09.398259 14670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:09.398388 14670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:42:09.406109 14670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.8.20:40275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:38111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:09.407701 14670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:09.409047 14670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:09.421545 14678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:09.422199 14676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:09.422796 14670 server_base.cc:1034] running on GCE node
W20260924 10:42:09.428131 14675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:09.428695 14670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:09.429530 14670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:09.445308 14670 hybrid_clock.cc:648] HybridClock initialized: now 1790246529445207 us; error 95 us; skew 500 ppm
Sep 24 10:42:09 dist-test-slave-j38h krb5kdc[13839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246529, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:09.449251 14670 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:42:09.450502 14670 webserver.cc:460] Webserver started at http://127.2.8.3:35041/ using document root <none> and password file <none>
I20260924 10:42:09.451237 14670 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:09.451313 14670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:09.451532 14670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 10:42:09 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246529, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:42:09.475615 14670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "353401a2b7db4883bb84dd50d2908673"
format_stamp: "Formatted at 2026-09-24 10:42:09 on dist-test-slave-j38h"
server_key: "eed8bf70b1bbf433ff7b3e2a0b332a53"
server_key_iv: "5410c586438a41e4598d9c1605a6ab4d"
server_key_version: "kuduclusterkey@0"
I20260924 10:42:09.476353 14670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "353401a2b7db4883bb84dd50d2908673"
format_stamp: "Formatted at 2026-09-24 10:42:09 on dist-test-slave-j38h"
server_key: "eed8bf70b1bbf433ff7b3e2a0b332a53"
server_key_iv: "5410c586438a41e4598d9c1605a6ab4d"
server_key_version: "kuduclusterkey@0"
I20260924 10:42:09.486901 14670 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.006s sys 0.001s
I20260924 10:42:09.525475 14685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:09.527575 14670 fs_manager.cc:725] Time spent opening block manager: real 0.025s user 0.002s sys 0.001s
I20260924 10:42:09.527750 14670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "353401a2b7db4883bb84dd50d2908673"
format_stamp: "Formatted at 2026-09-24 10:42:09 on dist-test-slave-j38h"
server_key: "eed8bf70b1bbf433ff7b3e2a0b332a53"
server_key_iv: "5410c586438a41e4598d9c1605a6ab4d"
server_key_version: "kuduclusterkey@0"
I20260924 10:42:09.527904 14670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:09.548126 14670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:09.551756 14670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:09.551994 14670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:09.552645 14670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:09.553665 14670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:42:09.553769 14670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:09.553858 14670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:09.553918 14670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:09.563534 14670 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:37403
I20260924 10:42:09.564705 14670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:42:09.568358 14798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:37403 every 8 connection(s)
I20260924 10:42:09.569552 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 14670
I20260924 10:42:09.569672 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:42:09.569962 2080 external_mini_cluster.cc:1428] Setting key c4f2955a9b91de19d551140021190079
Sep 24 10:42:09 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246529, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:09.586071 14529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:09.571367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:40795 (local address 127.2.8.62:38111)
0924 10:42:09.571611 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:09.571616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:09.573123 (+ 1507us) server_negotiation.cc:407] Connection header received
0924 10:42:09.574294 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:09.574298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:09.574347 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:09.574396 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:09.575978 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:09.576775 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:09.578434 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:09.578766 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:09.578843 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:09.582069 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:09.582072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:09.582075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:09.582112 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:09.583785 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:09.584390 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:09.584393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:09.584395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:09.584478 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:09.584844 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:09.584847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:09.584848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:09.585072 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:09.585162 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:09.585803 (+ 641us) server_negotiation.cc:299] Negotiation successful
0924 10:42:09.585918 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260924 10:42:09.587548 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:09.571795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38111 (local address 127.2.8.3:40795)
0924 10:42:09.572929 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:09.573014 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:09.574056 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:09.574799 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:09.574808 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:09.575160 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:09.575822 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:09.575840 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:09.576924 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:09.576927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:09.578305 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:09.578314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:09.578983 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:09.579037 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:09.579241 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:09.579922 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:09.579942 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:09.581881 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:09.583918 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:09.583924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:09.583939 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:09.584272 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:09.584578 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:09.584581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:09.584583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:09.584744 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:09.585225 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:09.585231 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:09.585542 (+ 311us) client_negotiation.cc:769] Sending connection context
0924 10:42:09.586234 (+ 692us) client_negotiation.cc:240] Negotiation successful
0924 10:42:09.586559 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":116,"threads_started":1}
I20260924 10:42:09.588858 14799 heartbeater.cc:347] Connected to a master server at 127.2.8.62:38111
I20260924 10:42:09.589118 14799 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:09.589666 14799 heartbeater.cc:511] Master 127.2.8.62:38111 requested a full tablet report, sending...
I20260924 10:42:09.590806 14315 ts_manager.cc:194] Registered new tserver with Master: 353401a2b7db4883bb84dd50d2908673 (127.2.8.3:37403)
I20260924 10:42:09.591543 14315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:40795
I20260924 10:42:09.596148 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:42:09.597013 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 14396
I20260924 10:42:09.603766 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 14535
I20260924 10:42:09.612327 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 14670
I20260924 10:42:09.619311 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 14283
I20260924 10:42:09.627583 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:38111
--webserver_interface=127.2.8.62
--webserver_port=35547
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:38111
--ranger_config_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.8.20:40275/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.8.20:40275/kms with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:09.738135 14802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:09.738379 14802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:09.738417 14802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:09.742482 14802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:42:09.742547 14802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:09.742568 14802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:09.742583 14802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:42:09.742599 14802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:42:09.748072 14802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.8.20:40275/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:38111
--ranger_config_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:38111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=35547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:09.749161 14802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:09.750183 14802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:09.758527 14810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:09.758560 14807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:09.758749 14802 server_base.cc:1034] running on GCE node
W20260924 10:42:09.758929 14808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:09.759521 14802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:09.760596 14802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:09.761755 14802 hybrid_clock.cc:648] HybridClock initialized: now 1790246529761742 us; error 28 us; skew 500 ppm
Sep 24 10:42:09 dist-test-slave-j38h krb5kdc[13839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246529, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:09.764413 14802 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:42:09.765316 14802 webserver.cc:460] Webserver started at http://127.2.8.62:35547/ using document root <none> and password file <none>
I20260924 10:42:09.765817 14802 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:09.765882 14802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:42:09 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246529, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:42:09.833771 14802 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 10:42:09.836155 14818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:09.837558 14802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:42:09.837671 14802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "510c721fd0344cd8b40bc79ec83d1c2d"
format_stamp: "Formatted at 2026-09-24 10:42:07 on dist-test-slave-j38h"
server_key: "6b11b606e235419b53626f31c97b843e"
server_key_iv: "c150127b3d13cd0351a74f82e6c3e586"
server_key_version: "kuduclusterkey@0"
I20260924 10:42:09.838207 14802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:09.870448 14802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:09.873607 14802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:09.873775 14802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:09.881219 14802 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:38111
I20260924 10:42:09.881232 14870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:38111 every 8 connection(s)
I20260924 10:42:09.882297 14802 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:42:09.886234 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 14802
I20260924 10:42:09.886776 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.8.20:40275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:35451
--local_ip_for_outbound_sockets=127.2.8.1
--tserver_master_addrs=127.2.8.62:38111
--webserver_port=46225
--webserver_interface=127.2.8.1
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.8.20:40275/kms with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:42:09.887419 14871 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:42:09.889504 14871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d: Bootstrap starting.
I20260924 10:42:09.907135 14871 log.cc:824] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:09.914199 14871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:42:09.914641 14871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d: Bootstrap complete.
I20260924 10:42:09.919179 14871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510c721fd0344cd8b40bc79ec83d1c2d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38111 } }
I20260924 10:42:09.919617 14871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510c721fd0344cd8b40bc79ec83d1c2d, State: Initialized, Role: FOLLOWER
I20260924 10:42:09.920125 14871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510c721fd0344cd8b40bc79ec83d1c2d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38111 } }
I20260924 10:42:09.920241 14871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:42:09.920341 14871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:42:09.920464 14871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:42:09.922577 14871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510c721fd0344cd8b40bc79ec83d1c2d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38111 } }
I20260924 10:42:09.922936 14871 leader_election.cc:302] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 510c721fd0344cd8b40bc79ec83d1c2d; no voters:
I20260924 10:42:09.923281 14871 leader_election.cc:288] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:42:09.923424 14875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:42:09.923684 14875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [term 2 LEADER]: Becoming Leader. State: Replica: 510c721fd0344cd8b40bc79ec83d1c2d, State: Running, Role: LEADER
I20260924 10:42:09.924050 14875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510c721fd0344cd8b40bc79ec83d1c2d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38111 } }
I20260924 10:42:09.924561 14871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:42:09.925429 14877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 510c721fd0344cd8b40bc79ec83d1c2d. Latest consensus state: current_term: 2 leader_uuid: "510c721fd0344cd8b40bc79ec83d1c2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510c721fd0344cd8b40bc79ec83d1c2d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38111 } } }
I20260924 10:42:09.925544 14877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:09.926278 14876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "510c721fd0344cd8b40bc79ec83d1c2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510c721fd0344cd8b40bc79ec83d1c2d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38111 } } }
I20260924 10:42:09.926367 14876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:09.926692 14883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:42:09.927076 14871 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 10:42:09.933153 14883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:42:09.933912 14883 catalog_manager.cc:1259] Loaded cluster ID: 71fc3c056e0043dbb05cee2244aa8980
I20260924 10:42:09.934005 14883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:42:09.936486 14883 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:42:09.937212 14883 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d: Loaded TSK: 0
I20260924 10:42:09.938479 14883 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:42:10.016719 14873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:10.017110 14873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:10.017210 14873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:10.021711 14873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:10.021790 14873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:10.021888 14873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:42:10.027532 14873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.8.20:40275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:35451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=46225
--tserver_master_addrs=127.2.8.62:38111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:10.028615 14873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:10.029981 14873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:10.041417 14912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:10.041519 14873 server_base.cc:1034] running on GCE node
W20260924 10:42:10.041698 14910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:10.041782 14909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:10.042445 14873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:10.043104 14873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:10.044271 14873 hybrid_clock.cc:648] HybridClock initialized: now 1790246530044202 us; error 87 us; skew 500 ppm
Sep 24 10:42:10 dist-test-slave-j38h krb5kdc[13839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:10.047102 14873 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:42:10.048048 14873 webserver.cc:460] Webserver started at http://127.2.8.1:46225/ using document root <none> and password file <none>
I20260924 10:42:10.048696 14873 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:10.048802 14873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:42:10 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:42:10.070461 14873 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260924 10:42:10.073655 14920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:10.075237 14873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:42:10.075358 14873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "f853cfd98be44b7fb1b80153045fcd2e"
format_stamp: "Formatted at 2026-09-24 10:42:08 on dist-test-slave-j38h"
server_key: "b8d3bd3499c186d22a6f4ec0b490b1b2"
server_key_iv: "ad7cb2cad655ea7cc89cfdf31649d647"
server_key_version: "kuduclusterkey@0"
I20260924 10:42:10.076073 14873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:10.103062 14873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:10.108618 14873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:10.108870 14873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:10.109728 14873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:10.110869 14873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:42:10.110948 14873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:10.111055 14873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:10.111094 14873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:10.122905 14873 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:35451
I20260924 10:42:10.123974 14873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:42:10.124737 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 14873
I20260924 10:42:10.125391 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.8.20:40275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:40517
--local_ip_for_outbound_sockets=127.2.8.2
--tserver_master_addrs=127.2.8.62:38111
--webserver_port=42291
--webserver_interface=127.2.8.2
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.8.20:40275/kms with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:42:10.133939 15033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:35451 every 8 connection(s)
Sep 24 10:42:10 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:10.154340 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:10.132796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38111 (local address 127.2.8.1:49935)
0924 10:42:10.134956 (+ 2160us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:10.135020 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:10.136343 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:10.138837 (+ 2494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:10.138849 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:10.139203 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:10.139834 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:10.139852 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:10.141967 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:10.141972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:10.143434 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:10.143447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:10.144690 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:10.144750 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:10.144941 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:10.146018 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:10.146046 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:10.147777 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:10.150573 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:10.150580 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:10.150596 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:10.150948 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:10.151397 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:10.151400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:10.151402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:10.151587 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:10.152363 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:10.152369 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:10.152698 (+ 329us) client_negotiation.cc:769] Sending connection context
0924 10:42:10.152899 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 10:42:10.153277 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1895,"thread_start_us":185,"threads_started":1}
I20260924 10:42:10.156406 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:10.132571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:49935 (local address 127.2.8.62:38111)
0924 10:42:10.136939 (+ 4368us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:10.136968 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:10.137060 (+ 92us) server_negotiation.cc:407] Connection header received
0924 10:42:10.137395 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:10.137412 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:10.137873 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:10.138261 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:10.140044 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:10.141801 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:10.143620 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:10.144350 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:10.144520 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:10.148024 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:10.148031 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:10.148051 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:10.148089 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:10.150326 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:10.151119 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:10.151125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:10.151131 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:10.151281 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:10.151707 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:10.151712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:10.151714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:10.152218 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:10.155079 (+ 2861us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:10.155236 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 10:42:10.155545 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4175,"thread_start_us":132,"threads_started":1}
I20260924 10:42:10.159070 15034 heartbeater.cc:347] Connected to a master server at 127.2.8.62:38111
I20260924 10:42:10.159302 15034 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:10.159873 15034 heartbeater.cc:511] Master 127.2.8.62:38111 requested a full tablet report, sending...
W20260924 10:42:10.163302 15034 heartbeater.cc:644] Failed to heartbeat to 127.2.8.62:38111 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:42:10.163424 15034 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:10.163623 15034 heartbeater.cc:511] Master 127.2.8.62:38111 requested a full tablet report, sending...
I20260924 10:42:10.164674 15034 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:10.164857 15034 heartbeater.cc:511] Master 127.2.8.62:38111 requested a full tablet report, sending...
W20260924 10:42:10.165774 15034 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:42:10.274816 15035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:10.275182 15035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:10.275254 15035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:10.280603 15035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:10.280710 15035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:10.280812 15035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:42:10.287428 15035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.8.20:40275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:40517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=42291
--tserver_master_addrs=127.2.8.62:38111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:10.288789 15035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:10.290181 15035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:10.302505 15044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:10.302510 15043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:10.302560 15046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:10.303069 15035 server_base.cc:1034] running on GCE node
I20260924 10:42:10.303489 15035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:10.304121 15035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:10.305292 15035 hybrid_clock.cc:648] HybridClock initialized: now 1790246530305260 us; error 46 us; skew 500 ppm
Sep 24 10:42:10 dist-test-slave-j38h krb5kdc[13839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:10.308051 15035 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:42:10.309115 15035 webserver.cc:460] Webserver started at http://127.2.8.2:42291/ using document root <none> and password file <none>
I20260924 10:42:10.309728 15035 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:10.309829 15035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:42:10 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:42:10.329394 15035 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 10:42:10.333990 15054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:10.335479 15035 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260924 10:42:10.335621 15035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "95825a87dcba45fe8ac7235fbcb76d16"
format_stamp: "Formatted at 2026-09-24 10:42:09 on dist-test-slave-j38h"
server_key: "cf2e4c5cb063cd3e5ead59d3b27b9abf"
server_key_iv: "3527e8d9fc9b946482e4f808a629b204"
server_key_version: "kuduclusterkey@0"
I20260924 10:42:10.336194 15035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:10.368461 15035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:10.372082 15035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:10.372352 15035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:10.373188 15035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:10.374249 15035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:42:10.374338 15035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:10.374416 15035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:10.374460 15035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:10.386921 15035 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:40517
I20260924 10:42:10.388242 15035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:42:10.394251 15167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:40517 every 8 connection(s)
I20260924 10:42:10.394488 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 15035
I20260924 10:42:10.395057 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.8.20:40275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:37403
--local_ip_for_outbound_sockets=127.2.8.3
--tserver_master_addrs=127.2.8.62:38111
--webserver_port=35041
--webserver_interface=127.2.8.3
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:40271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.8.20:40275/kms with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:42:10 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:10.413530 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:10.398255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:38055 (local address 127.2.8.62:38111)
0924 10:42:10.398388 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:10.398393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:10.400287 (+ 1894us) server_negotiation.cc:407] Connection header received
0924 10:42:10.401537 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:10.401541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:10.401603 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:10.401665 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:10.403380 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:10.404206 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:10.405946 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:10.406303 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:10.406380 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:10.409415 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:10.409420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:10.409422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:10.409451 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:10.411235 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:10.411850 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:10.411853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:10.411855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:10.411935 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:10.412336 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:10.412339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:10.412341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:10.412552 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:10.412628 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:10.413235 (+ 607us) server_negotiation.cc:299] Negotiation successful
0924 10:42:10.413356 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:42:10.414369 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:10.397145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38111 (local address 127.2.8.2:38055)
0924 10:42:10.400083 (+ 2938us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:10.400138 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:10.401273 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:10.402151 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:10.402160 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:10.402525 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:10.403190 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:10.403222 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:10.404368 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:10.404371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:10.405802 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:10.405813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:10.406527 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:10.406584 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:10.406793 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:10.407550 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:10.407571 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:10.409228 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:10.411367 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:10.411374 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:10.411389 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:10.411731 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:10.412042 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:10.412045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:10.412047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:10.412216 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:10.412691 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:10.412695 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:10.412921 (+ 226us) client_negotiation.cc:769] Sending connection context
0924 10:42:10.413168 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 10:42:10.413469 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2714,"thread_start_us":140,"threads_started":1}
I20260924 10:42:10.415577 15168 heartbeater.cc:347] Connected to a master server at 127.2.8.62:38111
I20260924 10:42:10.415828 15168 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:10.416314 15168 heartbeater.cc:511] Master 127.2.8.62:38111 requested a full tablet report, sending...
W20260924 10:42:10.417822 15168 heartbeater.cc:644] Failed to heartbeat to 127.2.8.62:38111 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:42:10.417950 15168 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:10.418143 15168 heartbeater.cc:511] Master 127.2.8.62:38111 requested a full tablet report, sending...
I20260924 10:42:10.418982 15168 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:10.419139 15168 heartbeater.cc:511] Master 127.2.8.62:38111 requested a full tablet report, sending...
W20260924 10:42:10.419864 15168 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:42:10.542330 15170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:10.542775 15170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:10.542906 15170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:10.548396 15170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:10.548488 15170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:10.548575 15170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:42:10.554157 15170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:40271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.8.20:40275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:37403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=35041
--tserver_master_addrs=127.2.8.62:38111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:10.555531 15170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:10.556932 15170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:10.567827 15178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:10.567889 15180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:10.568432 15177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:10.569830 15170 server_base.cc:1034] running on GCE node
I20260924 10:42:10.570258 15170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:10.571065 15170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:10.574435 15170 hybrid_clock.cc:648] HybridClock initialized: now 1790246530574392 us; error 47 us; skew 500 ppm
Sep 24 10:42:10 dist-test-slave-j38h krb5kdc[13839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:10.578224 15170 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:42:10.579346 15170 webserver.cc:460] Webserver started at http://127.2.8.3:35041/ using document root <none> and password file <none>
I20260924 10:42:10.580073 15170 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:10.580185 15170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 10:42:10 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:42:10.600879 15170 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:42:10.603844 15188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:10.605430 15170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 10:42:10.605551 15170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "353401a2b7db4883bb84dd50d2908673"
format_stamp: "Formatted at 2026-09-24 10:42:09 on dist-test-slave-j38h"
server_key: "eed8bf70b1bbf433ff7b3e2a0b332a53"
server_key_iv: "5410c586438a41e4598d9c1605a6ab4d"
server_key_version: "kuduclusterkey@0"
I20260924 10:42:10.606319 15170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:10.628062 15170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:10.631668 15170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:10.631877 15170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:10.632680 15170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:10.633776 15170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:42:10.633841 15170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:10.633896 15170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:10.633929 15170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:10.645385 15170 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:37403
I20260924 10:42:10.646725 15170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:42:10.647816 15301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:37403 every 8 connection(s)
I20260924 10:42:10.652894 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 15170
Sep 24 10:42:10 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246530, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:10.666978 15304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:10.649616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:38111 (local address 127.2.8.3:54691)
0924 10:42:10.650201 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:10.650243 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:10.651164 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:10.651710 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:10.651716 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:10.651977 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:10.652447 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:10.652460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:10.653354 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:10.653357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:10.654364 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:10.654374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:10.654918 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:10.654970 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:10.655187 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:10.656695 (+ 1508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:10.656712 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:10.658977 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:10.663575 (+ 4598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:10.663582 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:10.663601 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:10.663954 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:10.664262 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:10.664267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:10.664269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:10.664455 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:10.665013 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:10.665021 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:10.665368 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 10:42:10.665578 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 10:42:10.665929 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":137,"threads_started":1}
I20260924 10:42:10.667471 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:10.653635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45904 (local address 127.2.8.62:38111)
0924 10:42:10.654604 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:10.654608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:10.654618 (+ 10us) server_negotiation.cc:407] Connection header received
0924 10:42:10.654647 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:10.654649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:10.654690 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:10.654772 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:10.657108 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:10.657912 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:10.659182 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:10.659518 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:10.659638 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:10.661456 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:10.661460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:10.661463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:10.661490 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:10.664294 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:10.664785 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:10.664788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:10.664789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:10.664878 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:10.665206 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:10.665210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:10.665211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:10.665403 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:10.665607 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:10.667168 (+ 1561us) server_negotiation.cc:299] Negotiation successful
0924 10:42:10.667294 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"spinlock_wait_cycles":5504,"thread_start_us":89,"threads_started":1}
I20260924 10:42:10.667610 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:10.649195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:54691 (local address 127.2.8.62:38111)
0924 10:42:10.649316 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:10.649320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:10.650367 (+ 1047us) server_negotiation.cc:407] Connection header received
0924 10:42:10.651376 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:10.651379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:10.651437 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:10.651516 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:10.652616 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:10.653232 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:10.654488 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:10.654820 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:10.654918 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:10.661845 (+ 6927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:10.661850 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:10.661854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:10.661901 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:10.663433 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:10.664062 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:10.664067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:10.664068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:10.664171 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:10.664590 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:10.664593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:10.664595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:10.664838 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:10.664937 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:10.667390 (+ 2453us) server_negotiation.cc:299] Negotiation successful
0924 10:42:10.667481 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 10:42:10.668431 15302 heartbeater.cc:347] Connected to a master server at 127.2.8.62:38111
I20260924 10:42:10.668702 15302 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:10.669301 15302 heartbeater.cc:511] Master 127.2.8.62:38111 requested a full tablet report, sending...
W20260924 10:42:10.670864 15302 heartbeater.cc:644] Failed to heartbeat to 127.2.8.62:38111 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 10:42:10.671025 15302 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:10.671232 15302 heartbeater.cc:511] Master 127.2.8.62:38111 requested a full tablet report, sending...
I20260924 10:42:10.672065 15302 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:10.672236 15302 heartbeater.cc:511] Master 127.2.8.62:38111 requested a full tablet report, sending...
W20260924 10:42:10.673005 15302 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 10:42:11 dist-test-slave-j38h krb5kdc[13839](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790246531, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:11.166015 15034 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:11.166316 15034 heartbeater.cc:511] Master 127.2.8.62:38111 requested a full tablet report, sending...
I20260924 10:42:11.167613 14835 ts_manager.cc:194] Registered new tserver with Master: f853cfd98be44b7fb1b80153045fcd2e (127.2.8.1:35451)
I20260924 10:42:11.169407 14835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:49935
I20260924 10:42:11.420091 15168 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:11.420392 15168 heartbeater.cc:511] Master 127.2.8.62:38111 requested a full tablet report, sending...
I20260924 10:42:11.421300 14835 ts_manager.cc:194] Registered new tserver with Master: 95825a87dcba45fe8ac7235fbcb76d16 (127.2.8.2:40517)
I20260924 10:42:11.421912 14835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:38055
Sep 24 10:42:11 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246531, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:42:11.673280 15302 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:11.673605 15302 heartbeater.cc:511] Master 127.2.8.62:38111 requested a full tablet report, sending...
I20260924 10:42:11.674571 14835 ts_manager.cc:194] Registered new tserver with Master: 353401a2b7db4883bb84dd50d2908673 (127.2.8.3:37403)
I20260924 10:42:11.675379 14835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:54691
I20260924 10:42:11.679610 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:42:11.693490 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:11.682443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.2.8.62:38111)
0924 10:42:11.683188 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:11.683193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:11.683540 (+ 347us) server_negotiation.cc:407] Connection header received
0924 10:42:11.683587 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:11.683590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:11.683639 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:11.683737 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:11.684676 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.685597 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.686815 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.687175 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.687270 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:11.689814 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:11.689818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:11.689821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:11.689851 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:11.691661 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:11.692099 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:11.692104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:11.692106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:11.692193 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:11.692466 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:11.692471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:11.692472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:11.692663 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:11.692804 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:11.693075 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 10:42:11.693204 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":74,"threads_started":1}
I20260924 10:42:11.697635 14835 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:42:11.702594 14835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:42:11.726464 15332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:11.715139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:37403 (local address 127.0.0.1:33788)
0924 10:42:11.717956 (+ 2817us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:11.718002 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:11.718137 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:11.719296 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:11.719302 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:11.719319 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:11.719445 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:11.719451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.722547 (+ 3096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.722550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:11.723884 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:11.723896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.725884 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.725888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:11.726002 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:11.726042 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 10:42:11.726158 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 10:42:11.726285 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2631,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260924 10:42:11.727378 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:11.716731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.2.8.3:37403)
0924 10:42:11.718400 (+ 1669us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:11.718408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:11.718429 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:42:11.718502 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:11.718507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:11.718710 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:11.718852 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:11.719586 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.721975 (+ 2389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.724644 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.725515 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.725654 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:11.725814 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:11.727132 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0924 10:42:11.727240 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1568,"thread_start_us":92,"threads_started":1}
I20260924 10:42:11.730520 15236 tablet_service.cc:1467] Processing CreateTablet for tablet e6c9372b101c4ba19354f2b5d0f1555f (DEFAULT_TABLE table=test-table [id=957f9a9aec25429997b7f06438b52f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:42:11.731516 15236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c9372b101c4ba19354f2b5d0f1555f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:11.740417 15330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:11.716363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:35451 (local address 127.0.0.1:42768)
0924 10:42:11.719647 (+ 3284us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:11.719658 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:11.720669 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:11.721382 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:11.721385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:11.721401 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:11.721492 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:11.721498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.723579 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.723582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:11.735655 (+ 12073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:11.735670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.739861 (+ 4191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.739865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:11.739969 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:11.739997 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:42:11.740072 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 10:42:11.740168 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3226,"thread_start_us":73,"threads_started":1}
I20260924 10:42:11.740691 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:11.716536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42768 (local address 127.2.8.1:35451)
0924 10:42:11.720582 (+ 4046us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:11.720588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:11.720611 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:42:11.720855 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:11.720861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:11.721111 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:11.721237 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:11.721614 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.722780 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.737475 (+ 14695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.738400 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.738531 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:11.738714 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:11.740397 (+ 1683us) server_negotiation.cc:299] Negotiation successful
0924 10:42:11.740528 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3944,"thread_start_us":72,"threads_started":1}
I20260924 10:42:11.743680 14967 tablet_service.cc:1467] Processing CreateTablet for tablet e6c9372b101c4ba19354f2b5d0f1555f (DEFAULT_TABLE table=test-table [id=957f9a9aec25429997b7f06438b52f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:42:11.744275 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:11.716026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:40517 (local address 127.0.0.1:43564)
0924 10:42:11.719088 (+ 3062us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:11.719109 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:11.719188 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:11.737066 (+ 17878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:11.737069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:11.737092 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:11.737202 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:11.737208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.741730 (+ 4522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.741734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:11.742846 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:11.742857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.743899 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.743902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:11.743989 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:11.744013 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:42:11.744060 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 10:42:11.744129 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3001,"thread_start_us":103,"threads_started":1}
I20260924 10:42:11.744794 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c9372b101c4ba19354f2b5d0f1555f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:11.744938 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:11.716871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43564 (local address 127.2.8.2:40517)
0924 10:42:11.736079 (+ 19208us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:11.736086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:11.736112 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:42:11.736191 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:11.736196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:11.736396 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:11.736542 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:11.738485 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.739717 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.743003 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.743763 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.744526 (+ 763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:11.744676 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:11.744723 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 10:42:11.744810 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19080,"thread_start_us":66,"threads_started":1}
Sep 24 10:42:11 dist-test-slave-j38h krb5kdc[13839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246531, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for HTTP/127.2.8.20@KRBTEST.COM
I20260924 10:42:11.746683 15336 tablet_bootstrap.cc:492] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673: Bootstrap starting.
I20260924 10:42:11.747601 15102 tablet_service.cc:1467] Processing CreateTablet for tablet e6c9372b101c4ba19354f2b5d0f1555f (DEFAULT_TABLE table=test-table [id=957f9a9aec25429997b7f06438b52f9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:42:11.748553 15102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6c9372b101c4ba19354f2b5d0f1555f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:11.749020 15336 tablet_bootstrap.cc:654] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:11.749866 15336 log.cc:824] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:11.758602 15338 tablet_bootstrap.cc:492] T e6c9372b101c4ba19354f2b5d0f1555f P f853cfd98be44b7fb1b80153045fcd2e: Bootstrap starting.
I20260924 10:42:11.759281 15339 tablet_bootstrap.cc:492] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16: Bootstrap starting.
I20260924 10:42:11.761053 15338 tablet_bootstrap.cc:654] T e6c9372b101c4ba19354f2b5d0f1555f P f853cfd98be44b7fb1b80153045fcd2e: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:11.761525 15339 tablet_bootstrap.cc:654] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:11.761917 15338 log.cc:824] T e6c9372b101c4ba19354f2b5d0f1555f P f853cfd98be44b7fb1b80153045fcd2e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:11.762393 15339 log.cc:824] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:11.764180 15336 tablet_bootstrap.cc:492] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673: No bootstrap required, opened a new log
I20260924 10:42:11.764393 15336 ts_tablet_manager.cc:1397] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.003s
I20260924 10:42:11.764586 15339 tablet_bootstrap.cc:492] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16: No bootstrap required, opened a new log
I20260924 10:42:11.764755 15339 ts_tablet_manager.cc:1397] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260924 10:42:11.768667 15336 raft_consensus.cc:348] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353401a2b7db4883bb84dd50d2908673" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37403 } } peers { permanent_uuid: "95825a87dcba45fe8ac7235fbcb76d16" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40517 } } peers { permanent_uuid: "f853cfd98be44b7fb1b80153045fcd2e" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35451 } }
I20260924 10:42:11.768931 15336 raft_consensus.cc:374] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:11.769205 15336 raft_consensus.cc:729] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353401a2b7db4883bb84dd50d2908673, State: Initialized, Role: FOLLOWER
I20260924 10:42:11.769600 15339 raft_consensus.cc:348] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353401a2b7db4883bb84dd50d2908673" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37403 } } peers { permanent_uuid: "95825a87dcba45fe8ac7235fbcb76d16" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40517 } } peers { permanent_uuid: "f853cfd98be44b7fb1b80153045fcd2e" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35451 } }
I20260924 10:42:11.769784 15336 consensus_queue.cc:260] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353401a2b7db4883bb84dd50d2908673" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37403 } } peers { permanent_uuid: "95825a87dcba45fe8ac7235fbcb76d16" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40517 } } peers { permanent_uuid: "f853cfd98be44b7fb1b80153045fcd2e" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35451 } }
I20260924 10:42:11.769913 15339 raft_consensus.cc:374] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:11.769989 15339 raft_consensus.cc:729] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95825a87dcba45fe8ac7235fbcb76d16, State: Initialized, Role: FOLLOWER
I20260924 10:42:11.770627 15339 consensus_queue.cc:260] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353401a2b7db4883bb84dd50d2908673" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37403 } } peers { permanent_uuid: "95825a87dcba45fe8ac7235fbcb76d16" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40517 } } peers { permanent_uuid: "f853cfd98be44b7fb1b80153045fcd2e" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35451 } }
I20260924 10:42:11.771108 15336 ts_tablet_manager.cc:1428] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260924 10:42:11.771510 15302 heartbeater.cc:503] Master 127.2.8.62:38111 was elected leader, sending a full tablet report...
I20260924 10:42:11.771629 15339 ts_tablet_manager.cc:1428] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 10:42:11.771936 15168 heartbeater.cc:503] Master 127.2.8.62:38111 was elected leader, sending a full tablet report...
I20260924 10:42:11.774191 15338 tablet_bootstrap.cc:492] T e6c9372b101c4ba19354f2b5d0f1555f P f853cfd98be44b7fb1b80153045fcd2e: No bootstrap required, opened a new log
I20260924 10:42:11.774459 15338 ts_tablet_manager.cc:1397] T e6c9372b101c4ba19354f2b5d0f1555f P f853cfd98be44b7fb1b80153045fcd2e: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20260924 10:42:11.778759 15338 raft_consensus.cc:348] T e6c9372b101c4ba19354f2b5d0f1555f P f853cfd98be44b7fb1b80153045fcd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353401a2b7db4883bb84dd50d2908673" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37403 } } peers { permanent_uuid: "95825a87dcba45fe8ac7235fbcb76d16" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40517 } } peers { permanent_uuid: "f853cfd98be44b7fb1b80153045fcd2e" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35451 } }
I20260924 10:42:11.779160 15338 raft_consensus.cc:374] T e6c9372b101c4ba19354f2b5d0f1555f P f853cfd98be44b7fb1b80153045fcd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:11.779301 15338 raft_consensus.cc:729] T e6c9372b101c4ba19354f2b5d0f1555f P f853cfd98be44b7fb1b80153045fcd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f853cfd98be44b7fb1b80153045fcd2e, State: Initialized, Role: FOLLOWER
I20260924 10:42:11.779834 15338 consensus_queue.cc:260] T e6c9372b101c4ba19354f2b5d0f1555f P f853cfd98be44b7fb1b80153045fcd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353401a2b7db4883bb84dd50d2908673" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37403 } } peers { permanent_uuid: "95825a87dcba45fe8ac7235fbcb76d16" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40517 } } peers { permanent_uuid: "f853cfd98be44b7fb1b80153045fcd2e" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35451 } }
I20260924 10:42:11.781159 15338 ts_tablet_manager.cc:1428] T e6c9372b101c4ba19354f2b5d0f1555f P f853cfd98be44b7fb1b80153045fcd2e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 10:42:11.781569 15034 heartbeater.cc:503] Master 127.2.8.62:38111 was elected leader, sending a full tablet report...
W20260924 10:42:11.885883 15036 tablet.cc:2369] T e6c9372b101c4ba19354f2b5d0f1555f P f853cfd98be44b7fb1b80153045fcd2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:42:11.896730 15169 tablet.cc:2369] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:42:11.897684 15343 raft_consensus.cc:482] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:42:11.897884 15343 raft_consensus.cc:504] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353401a2b7db4883bb84dd50d2908673" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37403 } } peers { permanent_uuid: "95825a87dcba45fe8ac7235fbcb76d16" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40517 } } peers { permanent_uuid: "f853cfd98be44b7fb1b80153045fcd2e" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35451 } }
I20260924 10:42:11.899300 15343 leader_election.cc:288] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 353401a2b7db4883bb84dd50d2908673 (127.2.8.3:37403), f853cfd98be44b7fb1b80153045fcd2e (127.2.8.1:35451)
W20260924 10:42:11.899565 15303 tablet.cc:2369] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:42:11.905710 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:11.899993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:51171 (local address 127.2.8.3:37403)
0924 10:42:11.900155 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:11.900160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:11.901187 (+ 1027us) server_negotiation.cc:407] Connection header received
0924 10:42:11.901407 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:11.901411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:11.901466 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:11.901520 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:11.901982 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.902767 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.904368 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.904992 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.905087 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:11.905154 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:11.905499 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 10:42:11.905579 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:42:11.906091 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:11.899589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:37403 (local address 127.2.8.2:51171)
0924 10:42:11.901088 (+ 1499us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:11.901104 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:11.901270 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:11.901677 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:11.901680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:11.901700 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:11.901843 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:11.901850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.902912 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.902915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:11.904226 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:11.904236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.905249 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.905252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:11.905354 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:11.905378 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:42:11.905866 (+ 488us) client_negotiation.cc:240] Negotiation successful
0924 10:42:11.905941 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1419,"thread_start_us":121,"threads_started":1}
I20260924 10:42:11.906893 15256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c9372b101c4ba19354f2b5d0f1555f" candidate_uuid: "95825a87dcba45fe8ac7235fbcb76d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353401a2b7db4883bb84dd50d2908673" is_pre_election: true
I20260924 10:42:11.907207 15256 raft_consensus.cc:2454] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95825a87dcba45fe8ac7235fbcb76d16 in term 0.
I20260924 10:42:11.907720 15056 leader_election.cc:302] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353401a2b7db4883bb84dd50d2908673, 95825a87dcba45fe8ac7235fbcb76d16; no voters:
I20260924 10:42:11.908012 15343 raft_consensus.cc:2781] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:42:11.908118 15343 raft_consensus.cc:482] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:42:11.908167 15343 raft_consensus.cc:3037] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:11.909543 15343 raft_consensus.cc:504] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353401a2b7db4883bb84dd50d2908673" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37403 } } peers { permanent_uuid: "95825a87dcba45fe8ac7235fbcb76d16" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40517 } } peers { permanent_uuid: "f853cfd98be44b7fb1b80153045fcd2e" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35451 } }
I20260924 10:42:11.909907 15343 leader_election.cc:288] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [CANDIDATE]: Term 1 election: Requested vote from peers 353401a2b7db4883bb84dd50d2908673 (127.2.8.3:37403), f853cfd98be44b7fb1b80153045fcd2e (127.2.8.1:35451)
I20260924 10:42:11.910504 15256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c9372b101c4ba19354f2b5d0f1555f" candidate_uuid: "95825a87dcba45fe8ac7235fbcb76d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353401a2b7db4883bb84dd50d2908673"
I20260924 10:42:11.910624 15256 raft_consensus.cc:3037] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:11.911899 15256 raft_consensus.cc:2454] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95825a87dcba45fe8ac7235fbcb76d16 in term 1.
I20260924 10:42:11.912262 15056 leader_election.cc:302] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353401a2b7db4883bb84dd50d2908673, 95825a87dcba45fe8ac7235fbcb76d16; no voters:
I20260924 10:42:11.912501 15343 raft_consensus.cc:2781] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:42:11.912698 15343 raft_consensus.cc:686] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [term 1 LEADER]: Becoming Leader. State: Replica: 95825a87dcba45fe8ac7235fbcb76d16, State: Running, Role: LEADER
I20260924 10:42:11.913056 15343 consensus_queue.cc:237] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353401a2b7db4883bb84dd50d2908673" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37403 } } peers { permanent_uuid: "95825a87dcba45fe8ac7235fbcb76d16" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40517 } } peers { permanent_uuid: "f853cfd98be44b7fb1b80153045fcd2e" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35451 } }
I20260924 10:42:11.913877 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:11.900649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:39833 (local address 127.2.8.1:35451)
0924 10:42:11.900772 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:11.900776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:11.901474 (+ 698us) server_negotiation.cc:407] Connection header received
0924 10:42:11.901704 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:11.901708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:11.901762 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:11.901812 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:11.910284 (+ 8472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.911077 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.912515 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.913151 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.913236 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:11.913300 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:11.913670 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 10:42:11.913753 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:42:11.914362 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:11.900273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:35451 (local address 127.2.8.2:39833)
0924 10:42:11.901375 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:11.901390 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:11.901564 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:11.909968 (+ 8404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:11.909971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:11.909994 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:11.910126 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:11.910133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.911231 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.911234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:11.912364 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:11.912375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.913397 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.913400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:11.913499 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:11.913524 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:42:11.914144 (+ 620us) client_negotiation.cc:240] Negotiation successful
0924 10:42:11.914219 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1019,"thread_start_us":95,"threads_started":1}
I20260924 10:42:11.915155 14969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c9372b101c4ba19354f2b5d0f1555f" candidate_uuid: "95825a87dcba45fe8ac7235fbcb76d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f853cfd98be44b7fb1b80153045fcd2e" is_pre_election: true
I20260924 10:42:11.915436 14969 raft_consensus.cc:2454] T e6c9372b101c4ba19354f2b5d0f1555f P f853cfd98be44b7fb1b80153045fcd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95825a87dcba45fe8ac7235fbcb76d16 in term 0.
I20260924 10:42:11.915769 14970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6c9372b101c4ba19354f2b5d0f1555f" candidate_uuid: "95825a87dcba45fe8ac7235fbcb76d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f853cfd98be44b7fb1b80153045fcd2e"
I20260924 10:42:11.915879 14970 raft_consensus.cc:3037] T e6c9372b101c4ba19354f2b5d0f1555f P f853cfd98be44b7fb1b80153045fcd2e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:11.917148 14970 raft_consensus.cc:2454] T e6c9372b101c4ba19354f2b5d0f1555f P f853cfd98be44b7fb1b80153045fcd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95825a87dcba45fe8ac7235fbcb76d16 in term 1.
I20260924 10:42:11.919402 14835 catalog_manager.cc:5495] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95825a87dcba45fe8ac7235fbcb76d16 (127.2.8.2). New cstate: current_term: 1 leader_uuid: "95825a87dcba45fe8ac7235fbcb76d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353401a2b7db4883bb84dd50d2908673" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95825a87dcba45fe8ac7235fbcb76d16" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 40517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f853cfd98be44b7fb1b80153045fcd2e" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35451 } health_report { overall_health: UNKNOWN } } }
I20260924 10:42:11.963989 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:11.958819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.2.8.2:40517)
0924 10:42:11.958935 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:11.958940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:11.959131 (+ 191us) server_negotiation.cc:407] Connection header received
0924 10:42:11.959231 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:11.959234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:11.959307 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:11.959425 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:42:11.959734 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.960576 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.961879 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:11.962229 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:11.962339 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:11.962691 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:42:11.963290 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:42:11.963407 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:11.963767 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 10:42:11.963827 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 10:42:11.971200 14970 raft_consensus.cc:1261] T e6c9372b101c4ba19354f2b5d0f1555f P f853cfd98be44b7fb1b80153045fcd2e [term 1 FOLLOWER]: Refusing update from remote peer 95825a87dcba45fe8ac7235fbcb76d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:42:11.971370 15256 raft_consensus.cc:1261] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673 [term 1 FOLLOWER]: Refusing update from remote peer 95825a87dcba45fe8ac7235fbcb76d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:42:11.972101 15348 consensus_queue.cc:1035] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "353401a2b7db4883bb84dd50d2908673" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:42:11.972306 15343 consensus_queue.cc:1035] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f853cfd98be44b7fb1b80153045fcd2e" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 35451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:42:11.989475 15352 mvcc.cc:205] Tried to move back new op lower bound from 7332849794946158592 to 7332849794719203328. Current Snapshot: MvccSnapshot[applied={T|T < 7332849794946158592}]
I20260924 10:42:11.995795 15354 mvcc.cc:205] Tried to move back new op lower bound from 7332849794946158592 to 7332849794719203328. Current Snapshot: MvccSnapshot[applied={T|T < 7332849794946158592 or (T in {7332849794946158592})}]
I20260924 10:42:11.996013 15351 mvcc.cc:205] Tried to move back new op lower bound from 7332849794946158592 to 7332849794719203328. Current Snapshot: MvccSnapshot[applied={T|T < 7332849794946158592 or (T in {7332849794946158592})}]
I20260924 10:42:11.998221 14835 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:42:11.998389 14835 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:42:12.001371 14835 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 510c721fd0344cd8b40bc79ec83d1c2d: Sending DeleteTablet for 3 replicas of tablet e6c9372b101c4ba19354f2b5d0f1555f
I20260924 10:42:12.002607 15236 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c9372b101c4ba19354f2b5d0f1555f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:42:11 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:33788
I20260924 10:42:12.003515 15102 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c9372b101c4ba19354f2b5d0f1555f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:42:11 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:43564
I20260924 10:42:12.004207 14967 tablet_service.cc:1514] Processing DeleteTablet for tablet e6c9372b101c4ba19354f2b5d0f1555f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:42:11 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:42768
I20260924 10:42:12.005311 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 14873
I20260924 10:42:12.009770 15361 tablet_replica.cc:330] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673: stopping tablet replica
I20260924 10:42:12.010015 15362 tablet_replica.cc:330] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16: stopping tablet replica
I20260924 10:42:12.012521 15362 raft_consensus.cc:2229] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:42:12.012916 15361 raft_consensus.cc:2229] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:42:12.013396 15362 raft_consensus.cc:2258] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:42:12.013804 15361 raft_consensus.cc:2258] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:42:12.015992 15362 ts_tablet_manager.cc:1905] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:42:12.016963 15361 ts_tablet_manager.cc:1905] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:42:12.023062 15361 ts_tablet_manager.cc:1918] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:42:12.025882 15362 ts_tablet_manager.cc:1918] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260924 10:42:12.026261 14819 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.8.1:35451 (error 108)
I20260924 10:42:12.027248 15362 log.cc:1196] T e6c9372b101c4ba19354f2b5d0f1555f P 95825a87dcba45fe8ac7235fbcb76d16: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/e6c9372b101c4ba19354f2b5d0f1555f
I20260924 10:42:12.027513 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 15035
I20260924 10:42:12.029277 15361 log.cc:1196] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/e6c9372b101c4ba19354f2b5d0f1555f
I20260924 10:42:12.029896 15361 ts_tablet_manager.cc:1939] T e6c9372b101c4ba19354f2b5d0f1555f P 353401a2b7db4883bb84dd50d2908673: Deleting consensus metadata
I20260924 10:42:12.033699 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:12.033353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:35451 (local address 127.0.0.1:42778)
0924 10:42:12.033543 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:12.033612 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:35451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260924 10:42:12.033939 14819 catalog_manager.cc:4584] TS f853cfd98be44b7fb1b80153045fcd2e (127.2.8.1:35451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6c9372b101c4ba19354f2b5d0f1555f: Network error: Client connection negotiation failed: client connection to 127.2.8.1:35451: connect: Connection refused (error 111)
I20260924 10:42:12.035251 14820 catalog_manager.cc:4841] TS 353401a2b7db4883bb84dd50d2908673 (127.2.8.3:37403): tablet e6c9372b101c4ba19354f2b5d0f1555f (table test-table [id=957f9a9aec25429997b7f06438b52f9f]) successfully deleted
I20260924 10:42:12.035835 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 15170
I20260924 10:42:12.035921 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:12.035638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:40517 (local address 127.0.0.1:43582)
0924 10:42:12.035798 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:12.035844 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.2:40517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 10:42:12.041934 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 14802
I20260924 10:42:12.047362 2080 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 10:42:12.382323 2080 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 10:42:12.382459 2080 mini_ranger.cc:66] Stopping Ranger...
I20260924 10:42:12.727078 2080 mini_ranger.cc:68] Stopped Ranger
2026-09-24 10:42:12.727 UTC [13869] LOG: received smart shutdown request
2026-09-24 10:42:12.731 UTC [13869] LOG: background worker "logical replication launcher" (PID 13878) exited with exit code 1
2026-09-24 10:42:12.732 UTC [13873] LOG: shutting down
2026-09-24 10:42:12.911 UTC [13869] LOG: database system is shut down
2026-09-24T10:42:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49086 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:42:13 dist-test-slave-j38h krb5kdc[15375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:42:13 dist-test-slave-j38h krb5kdc[15375](info): set up 2 sockets
Sep 24 10:42:13 dist-test-slave-j38h krb5kdc[15375](info): commencing operation
krb5kdc: starting...
W20260924 10:42:14.073066 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:42:14 dist-test-slave-j38h krb5kdc[15375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:42:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:42:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:42:14.200614 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:45925
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:45925,127.2.8.61:35817,127.2.8.60:37069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:14.308700 15388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:14.308948 15388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:14.309005 15388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:14.312958 15388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:42:14.313026 15388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:14.313046 15388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:14.313062 15388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:42:14.313077 15388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:42:14.318200 15388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:45925,127.2.8.61:35817,127.2.8.60:37069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:45925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:14.319139 15388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:14.320142 15388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:14.328428 15396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:14.328431 15393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:14.328431 15394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:14.328603 15388 server_base.cc:1034] running on GCE node
I20260924 10:42:14.329290 15388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:14.330375 15388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:14.331535 15388 hybrid_clock.cc:648] HybridClock initialized: now 1790246534331517 us; error 30 us; skew 500 ppm
Sep 24 10:42:14 dist-test-slave-j38h krb5kdc[15375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246534, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:14.334317 15388 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:42:14.335108 15388 webserver.cc:460] Webserver started at http://127.2.8.62:43071/ using document root <none> and password file <none>
I20260924 10:42:14.335640 15388 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:14.335705 15388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:14.335868 15388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:42:14.337853 15388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "144b387f7b214adab97adab6c0982acc"
format_stamp: "Formatted at 2026-09-24 10:42:14 on dist-test-slave-j38h"
server_key: "d1ad39ba896d8eed51737be742ee18c6"
server_key_iv: "15719ba4a0372ffb4b13e946ef400e78"
server_key_version: "encryptionkey@0"
I20260924 10:42:14.338353 15388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "144b387f7b214adab97adab6c0982acc"
format_stamp: "Formatted at 2026-09-24 10:42:14 on dist-test-slave-j38h"
server_key: "d1ad39ba896d8eed51737be742ee18c6"
server_key_iv: "15719ba4a0372ffb4b13e946ef400e78"
server_key_version: "encryptionkey@0"
I20260924 10:42:14.342204 15388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:42:14.344524 15403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:14.345803 15388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:42:14.345898 15388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "144b387f7b214adab97adab6c0982acc"
format_stamp: "Formatted at 2026-09-24 10:42:14 on dist-test-slave-j38h"
server_key: "d1ad39ba896d8eed51737be742ee18c6"
server_key_iv: "15719ba4a0372ffb4b13e946ef400e78"
server_key_version: "encryptionkey@0"
I20260924 10:42:14.345981 15388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:14.373328 15388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:14.376510 15388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:14.376674 15388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:14.383569 15388 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:45925
I20260924 10:42:14.383630 15455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:45925 every 8 connection(s)
I20260924 10:42:14.384539 15388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:42:14.386580 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 15388
I20260924 10:42:14.386767 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:42:14.387076 2080 external_mini_cluster.cc:1428] Setting key fb871390a347a4c77b5951cd68c432ec
I20260924 10:42:14.387776 15456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:14.391743 15456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.8.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.61@KRBTEST.COM" created.
Sep 24 10:42:14 dist-test-slave-j38h krb5kdc[15375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246534, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:14.408409 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.393970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45925 (local address 127.0.0.1:47852)
0924 10:42:14.394441 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.394491 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:14.395311 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:14.396059 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:14.396064 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:14.396376 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:14.396613 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.396674 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.398234 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.398240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.399450 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.399460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.400348 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.400400 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.400591 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.401322 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:14.401341 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:14.403195 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:14.405695 (+ 2500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.405703 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.405711 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.405944 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.406315 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.406318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.406319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.406428 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.407049 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:14.407054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:14.407180 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 10:42:14.407323 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 10:42:14.407754 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":5,"thread_start_us":151,"threads_started":1}
I20260924 10:42:14.408421 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.393971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47852 (local address 127.2.8.62:45925)
0924 10:42:14.394537 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:14.394548 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:14.394594 (+ 46us) server_negotiation.cc:407] Connection header received
0924 10:42:14.395596 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:14.395602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:14.395852 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:14.396012 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:14.396954 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.398108 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.399571 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.400200 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.400399 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.403362 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:14.403369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:14.403385 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:14.403417 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:14.405563 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.406065 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.406075 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.406078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.406226 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.406511 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.406514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.406515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.406935 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:14.407038 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:14.407398 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 10:42:14.407556 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":189,"threads_started":1}
I20260924 10:42:14.410943 15456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:42:14.411813 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.411428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:35817 (local address 127.0.0.1:55442)
0924 10:42:14.411680 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.411750 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
W20260924 10:42:14.412102 15404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.8.61:35817: connect: Connection refused (error 111)
I20260924 10:42:14.413255 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.412949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:35817 (local address 127.0.0.1:55458)
0924 10:42:14.413154 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.413203 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260924 10:42:14.413452 15456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.61:35817: Network error: Client connection negotiation failed: client connection to 127.2.8.61:35817: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.61.keytab.
Entry for principal kudu/127.2.8.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.61.keytab.
I20260924 10:42:14.416911 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.61:35817
--webserver_interface=127.2.8.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.61.keytab
--principal=kudu/127.2.8.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:45925,127.2.8.61:35817,127.2.8.60:37069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:42:14.462779 15456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } attempt: 1
I20260924 10:42:14.463770 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.463425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:35817 (local address 127.0.0.1:55468)
0924 10:42:14.463612 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.463676 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 10:42:14.464550 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.464338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:35817 (local address 127.0.0.1:55480)
0924 10:42:14.464445 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.464491 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 10:42:14.464731 15456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.61:35817: Network error: Client connection negotiation failed: client connection to 127.2.8.61:35817: connect: Connection refused (error 111)
W20260924 10:42:14.527704 15463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:14.527940 15463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:14.527978 15463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:14.531875 15463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:42:14.531924 15463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:14.531944 15463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:14.531960 15463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:42:14.531975 15463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:42:14.532898 15456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } attempt: 2
I20260924 10:42:14.533743 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.533442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:35817 (local address 127.0.0.1:55494)
0924 10:42:14.533586 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.533649 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 10:42:14.534530 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.534237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:35817 (local address 127.0.0.1:55502)
0924 10:42:14.534412 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.534459 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:35817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 10:42:14.534680 15456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.61:35817: Network error: Client connection negotiation failed: client connection to 127.2.8.61:35817: connect: Connection refused (error 111)
I20260924 10:42:14.537634 15463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:45925,127.2.8.61:35817,127.2.8.60:37069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.61.keytab
--principal=kudu/127.2.8.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.61:35817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.8.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:14.538662 15463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:14.539640 15463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:14.547859 15471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:14.548018 15463 server_base.cc:1034] running on GCE node
W20260924 10:42:14.547842 15468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:14.547842 15469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:14.548664 15463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:14.549952 15463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:14.551132 15463 hybrid_clock.cc:648] HybridClock initialized: now 1790246534551112 us; error 37 us; skew 500 ppm
Sep 24 10:42:14 dist-test-slave-j38h krb5kdc[15375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246534, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:14.553960 15463 init.cc:377] Logged in from keytab as kudu/127.2.8.61@KRBTEST.COM (short username kudu)
I20260924 10:42:14.554756 15463 webserver.cc:460] Webserver started at http://127.2.8.61:40533/ using document root <none> and password file <none>
I20260924 10:42:14.555255 15463 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:14.555310 15463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:14.555467 15463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:42:14.557469 15463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/data/instance:
uuid: "c2cdef7d694e43a482a547994e63c5f5"
format_stamp: "Formatted at 2026-09-24 10:42:14 on dist-test-slave-j38h"
server_key: "6f61473c2b1e4e9dda75574d0c8e24f6"
server_key_iv: "0a8ebc7db0c2e5d4fd4700e726f54445"
server_key_version: "encryptionkey@0"
I20260924 10:42:14.557976 15463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/wal/instance:
uuid: "c2cdef7d694e43a482a547994e63c5f5"
format_stamp: "Formatted at 2026-09-24 10:42:14 on dist-test-slave-j38h"
server_key: "6f61473c2b1e4e9dda75574d0c8e24f6"
server_key_iv: "0a8ebc7db0c2e5d4fd4700e726f54445"
server_key_version: "encryptionkey@0"
I20260924 10:42:14.561561 15463 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:42:14.564002 15478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:14.565259 15463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:42:14.565362 15463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/wal
uuid: "c2cdef7d694e43a482a547994e63c5f5"
format_stamp: "Formatted at 2026-09-24 10:42:14 on dist-test-slave-j38h"
server_key: "6f61473c2b1e4e9dda75574d0c8e24f6"
server_key_iv: "0a8ebc7db0c2e5d4fd4700e726f54445"
server_key_version: "encryptionkey@0"
I20260924 10:42:14.565446 15463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:14.589550 15463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:14.592414 15463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:14.592563 15463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:14.600402 15463 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.61:35817
I20260924 10:42:14.600414 15530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.61:35817 every 8 connection(s)
I20260924 10:42:14.601388 15463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/data/info.pb
I20260924 10:42:14.603288 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 15463
I20260924 10:42:14.603392 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-1/wal/instance
I20260924 10:42:14.603706 2080 external_mini_cluster.cc:1428] Setting key 454b6d16013464b7f05f7d6726a40edc
I20260924 10:42:14.604197 15531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:14.607795 15531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.8.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.60@KRBTEST.COM" created.
Sep 24 10:42:14 dist-test-slave-j38h krb5kdc[15375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246534, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.61@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:14.621415 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.609645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.2.8.62:45925)
0924 10:42:14.609777 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:14.609781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:14.610738 (+ 957us) server_negotiation.cc:407] Connection header received
0924 10:42:14.611714 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:14.611718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:14.611781 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:14.611876 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:14.613035 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.613651 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.614781 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.615163 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.615277 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.617769 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:14.617772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:14.617774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:14.617797 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:14.619288 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.619792 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.619801 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.619804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.619872 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.620203 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.620209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.620211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.620484 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:14.620599 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:14.621114 (+ 515us) server_negotiation.cc:299] Negotiation successful
0924 10:42:14.621247 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:42:14.622247 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.610113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45925 (local address 127.0.0.1:47858)
0924 10:42:14.610591 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.610631 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:14.611472 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:14.612033 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:14.612040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:14.612389 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:14.612834 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.612847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.613761 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.613765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.614656 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.614665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.615275 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.615324 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.615466 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.616138 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:14.616153 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:14.617591 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:14.619407 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.619413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.619426 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.619682 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.619961 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.619966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.619968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.620101 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.620608 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:14.620616 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:14.620852 (+ 236us) client_negotiation.cc:769] Sending connection context
0924 10:42:14.621071 (+ 219us) client_negotiation.cc:240] Negotiation successful
0924 10:42:14.621363 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":143,"threads_started":1}
I20260924 10:42:14.623571 15531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:42:14.625824 15456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } attempt: 3
Sep 24 10:42:14 dist-test-slave-j38h krb5kdc[15375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246534, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.61@KRBTEST.COM for kudu/127.2.8.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.60.keytab.
Entry for principal kudu/127.2.8.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.60.keytab.
I20260924 10:42:14.630267 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.60:37069
--webserver_interface=127.2.8.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.60.keytab
--principal=kudu/127.2.8.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:45925,127.2.8.61:35817,127.2.8.60:37069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:42:14 dist-test-slave-j38h krb5kdc[15375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246534, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for kudu/127.2.8.61@KRBTEST.COM
I20260924 10:42:14.634249 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.624145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.2.8.61:35817)
0924 10:42:14.624464 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:14.624477 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:14.624501 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:42:14.624537 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:14.624540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:14.624688 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:14.624792 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:14.625183 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.626115 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.627156 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.627493 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.627572 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.629707 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:14.629721 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:14.629731 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:14.629749 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:14.631861 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.632289 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.632295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.632296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.632379 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.632714 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.632720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.632722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.633096 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:14.633255 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:14.633805 (+ 550us) server_negotiation.cc:299] Negotiation successful
0924 10:42:14.633968 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":153,"threads_started":1}
I20260924 10:42:14.634791 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.624035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:35817 (local address 127.0.0.1:55512)
0924 10:42:14.624164 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.624176 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:14.624236 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:14.624813 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:14.624815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:14.624939 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:14.625084 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.625090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.626229 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.626235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.627017 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.627024 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.627592 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.627607 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.627701 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.628128 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:14.628141 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:14.629608 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:14.631996 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.631999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.632001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.632175 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.632483 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.632489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.632492 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.632606 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.633467 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:14.633473 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:14.633587 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 10:42:14.634393 (+ 806us) client_negotiation.cc:240] Negotiation successful
0924 10:42:14.634582 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:42:14.636402 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.626187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:35817 (local address 127.0.0.1:55516)
0924 10:42:14.626348 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.626365 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:14.626453 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:14.626798 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:14.626800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:14.626924 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:14.627009 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.627014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.628117 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.628119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.629481 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.629492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.630103 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.630122 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.630244 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.630813 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:14.630831 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:14.632252 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:14.634880 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.634885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.634887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.635153 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.635427 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.635430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.635432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.635513 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.635858 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:14.635861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:14.635999 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 10:42:14.636109 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 10:42:14.636238 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 10:42:14.636420 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.626245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.2.8.61:35817)
0924 10:42:14.626574 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:14.626583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:14.626595 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:42:14.626626 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:14.626629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:14.626665 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:14.626771 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:14.627114 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.627790 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.629604 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.629979 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.630098 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.632365 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:14.632368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:14.632371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:14.632389 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:14.634772 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.635249 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.635252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.635253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.635330 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.635616 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.635618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.635619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.635761 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:14.635866 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:14.636158 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 10:42:14.636269 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":142,"threads_started":1}
I20260924 10:42:14.637189 15531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:42:14.637660 15456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:42:14.637913 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.637682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:37069 (local address 127.0.0.1:45842)
0924 10:42:14.637787 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.637865 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 10:42:14.638332 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.638112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:37069 (local address 127.0.0.1:45846)
0924 10:42:14.638221 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.638272 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:42:14.638341 15479 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
I20260924 10:42:14.639149 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.638751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:37069 (local address 127.0.0.1:45862)
0924 10:42:14.639034 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.639086 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":80,"threads_started":1}
I20260924 10:42:14.639251 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.638977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:37069 (local address 127.0.0.1:45870)
0924 10:42:14.639107 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.639175 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 10:42:14.639276 15456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:37069: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
W20260924 10:42:14.639410 15531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:37069: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
I20260924 10:42:14.670424 15456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } attempt: 1
I20260924 10:42:14.671346 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.671028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:37069 (local address 127.0.0.1:45880)
0924 10:42:14.671201 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.671270 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 10:42:14.672075 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.671893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:37069 (local address 127.0.0.1:45884)
0924 10:42:14.671985 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.672027 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 10:42:14.672216 15456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:37069: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
I20260924 10:42:14.688694 15531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } attempt: 1
I20260924 10:42:14.689622 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.689289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:37069 (local address 127.0.0.1:45900)
0924 10:42:14.689458 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.689535 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 10:42:14.690407 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.690167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:37069 (local address 127.0.0.1:45904)
0924 10:42:14.690288 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.690341 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 10:42:14.690544 15531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:37069: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
W20260924 10:42:14.740693 15538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:14.740934 15538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:14.741042 15538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:14.745213 15538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:42:14.745260 15538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:14.745278 15538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:14.745319 15538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:42:14.745348 15538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:42:14.747368 15456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } attempt: 2
I20260924 10:42:14.748162 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.747884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:37069 (local address 127.0.0.1:45906)
0924 10:42:14.748025 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.748088 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 10:42:14.748888 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.748639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:37069 (local address 127.0.0.1:45922)
0924 10:42:14.748748 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.748797 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 10:42:14.749032 15456 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:37069: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
I20260924 10:42:14.751026 15538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:45925,127.2.8.61:35817,127.2.8.60:37069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.60.keytab
--principal=kudu/127.2.8.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.60:37069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.8.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:14.752034 15538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:14.753088 15538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:42:14.758688 15531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } attempt: 2
I20260924 10:42:14.759418 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.759156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:37069 (local address 127.0.0.1:45938)
0924 10:42:14.759297 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.759352 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 10:42:14.760089 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.759869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:37069 (local address 127.0.0.1:45944)
0924 10:42:14.759973 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.760025 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 10:42:14.760368 15531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:37069: Network error: Client connection negotiation failed: client connection to 127.2.8.60:37069: connect: Connection refused (error 111)
W20260924 10:42:14.761358 15548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:14.761400 15545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:14.761409 15546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:14.761781 15538 server_base.cc:1034] running on GCE node
I20260924 10:42:14.762293 15538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:14.763375 15538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:14.764567 15538 hybrid_clock.cc:648] HybridClock initialized: now 1790246534764559 us; error 29 us; skew 500 ppm
Sep 24 10:42:14 dist-test-slave-j38h krb5kdc[15375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246534, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:14.767277 15538 init.cc:377] Logged in from keytab as kudu/127.2.8.60@KRBTEST.COM (short username kudu)
I20260924 10:42:14.768115 15538 webserver.cc:460] Webserver started at http://127.2.8.60:40739/ using document root <none> and password file <none>
I20260924 10:42:14.768669 15538 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:14.768728 15538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:14.768883 15538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:42:14.770891 15538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/data/instance:
uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80"
format_stamp: "Formatted at 2026-09-24 10:42:14 on dist-test-slave-j38h"
server_key: "c969afbd4a12495fb3eb63c9a92c3952"
server_key_iv: "d019653026c145caee56ad4dcbd23edd"
server_key_version: "encryptionkey@0"
I20260924 10:42:14.771399 15538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/wal/instance:
uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80"
format_stamp: "Formatted at 2026-09-24 10:42:14 on dist-test-slave-j38h"
server_key: "c969afbd4a12495fb3eb63c9a92c3952"
server_key_iv: "d019653026c145caee56ad4dcbd23edd"
server_key_version: "encryptionkey@0"
I20260924 10:42:14.774891 15538 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:42:14.777104 15555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:14.778266 15538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:42:14.778368 15538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/wal
uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80"
format_stamp: "Formatted at 2026-09-24 10:42:14 on dist-test-slave-j38h"
server_key: "c969afbd4a12495fb3eb63c9a92c3952"
server_key_iv: "d019653026c145caee56ad4dcbd23edd"
server_key_version: "encryptionkey@0"
I20260924 10:42:14.778446 15538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:14.806726 15538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:14.810312 15538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:14.810463 15538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:14.817322 15538 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.60:37069
I20260924 10:42:14.817358 15607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.60:37069 every 8 connection(s)
I20260924 10:42:14.818244 15538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/data/info.pb
I20260924 10:42:14.821108 15608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:14.824703 15608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:42:14.827018 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 15538
I20260924 10:42:14.827162 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790246374820021-2080-0/minicluster-data/master-2/wal/instance
I20260924 10:42:14.827443 2080 external_mini_cluster.cc:1428] Setting key e34385976038637599c149e383061378
Sep 24 10:42:14 dist-test-slave-j38h krb5kdc[15375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246534, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.60@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
Sep 24 10:42:14 dist-test-slave-j38h krb5kdc[15375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.60@KRBTEST.COM
I20260924 10:42:14.838760 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.826564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47872 (local address 127.2.8.62:45925)
0924 10:42:14.826720 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:14.826725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:14.827622 (+ 897us) server_negotiation.cc:407] Connection header received
0924 10:42:14.828828 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:14.828831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:14.828877 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:14.828954 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:14.829923 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.830437 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.831536 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.831867 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.831949 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.834670 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:14.834672 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:14.834675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:14.834698 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:14.836335 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.837049 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.837055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.837057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.837140 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.837525 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.837530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.837532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.837760 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:14.837874 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:14.838397 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 10:42:14.838520 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:42:14.839563 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.826966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45925 (local address 127.0.0.1:47872)
0924 10:42:14.827460 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.827511 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:14.828559 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:14.829043 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:14.829052 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:14.829488 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:14.829751 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.829765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.830547 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.830550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.831409 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.831417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.831956 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.832003 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.832170 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.832827 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:14.832850 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:14.834446 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:14.836461 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.836469 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.836478 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.836864 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.837271 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.837275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.837277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.837424 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.837871 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:14.837879 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:14.838146 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 10:42:14.838329 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 10:42:14.838652 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":160,"threads_started":1}
I20260924 10:42:14.839573 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.828180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.2.8.60:37069)
0924 10:42:14.828504 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:14.828515 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:14.828557 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:42:14.828740 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:14.828752 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:14.828986 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:14.829184 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:14.829850 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.830845 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.831767 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.832213 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.832320 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.835012 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:14.835023 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:14.835032 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:14.835051 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:14.837041 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.837503 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.837508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.837510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.837594 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.837878 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.837882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.837884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.838169 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:14.838374 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:14.838586 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 10:42:14.838743 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"spinlock_wait_cycles":1792,"thread_start_us":112,"threads_started":1}
I20260924 10:42:14.841130 15608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:42:14 dist-test-slave-j38h krb5kdc[15375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246534, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.60@KRBTEST.COM for kudu/127.2.8.61@KRBTEST.COM
Sep 24 10:42:14 dist-test-slave-j38h krb5kdc[15375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:14.848205 15456 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } attempt: 3
I20260924 10:42:14.851003 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.841580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:35817 (local address 127.0.0.1:55522)
0924 10:42:14.841702 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.841731 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:14.841810 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:14.842098 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:14.842101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:14.842289 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:14.842406 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.842413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.843278 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.843282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.843857 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.843867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.844419 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.844433 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.844531 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.845104 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:14.845114 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:14.846436 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:14.848386 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.848392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.848397 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.849268 (+ 871us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.849975 (+ 707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.849977 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.849979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.850053 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.850526 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:14.850528 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:14.850612 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 10:42:14.850715 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 10:42:14.850852 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 10:42:14.851068 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.841754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.2.8.61:35817)
0924 10:42:14.841878 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:14.841885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:14.841899 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:42:14.841940 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:14.841943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:14.841993 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:14.842085 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:14.842538 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.843166 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.844001 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.844317 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.844403 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.846656 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:14.846659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:14.846661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:14.846684 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:14.848272 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.849375 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.849382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.849384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.849525 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.850189 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.850193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.850194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.850405 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:14.850505 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:14.850782 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 10:42:14.850895 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 10:42:14.851512 15531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } attempt: 3
I20260924 10:42:14.851770 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.842301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.2.8.62:45925)
0924 10:42:14.842424 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:14.842426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:14.842523 (+ 97us) server_negotiation.cc:407] Connection header received
0924 10:42:14.842615 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:14.842618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:14.842659 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:14.842735 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:14.843289 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.843748 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.844769 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.845193 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.845293 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.847590 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:14.847595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:14.847599 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:14.847628 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:14.849520 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.849964 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.849967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.849969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.850047 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.850357 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.850361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.850363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.850609 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:14.850813 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:14.851458 (+ 645us) server_negotiation.cc:299] Negotiation successful
0924 10:42:14.851575 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 10:42:14 dist-test-slave-j38h krb5kdc[15375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246534, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.61@KRBTEST.COM for kudu/127.2.8.60@KRBTEST.COM
Sep 24 10:42:14 dist-test-slave-j38h krb5kdc[15375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.61@KRBTEST.COM
I20260924 10:42:14.866065 15608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:42:14 dist-test-slave-j38h krb5kdc[15375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246534, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for kudu/127.2.8.60@KRBTEST.COM
Sep 24 10:42:14 dist-test-slave-j38h krb5kdc[15375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246534, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.60@KRBTEST.COM for kudu/127.2.8.60@KRBTEST.COM
I20260924 10:42:14.873498 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.852621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:37069 (local address 127.0.0.1:45978)
0924 10:42:14.852778 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.852790 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:14.852896 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:14.853305 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:14.853312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:14.853526 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:14.853638 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.853647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.854896 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.854901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.855784 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.855796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.856481 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.856499 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.856592 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.857343 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:14.857360 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:14.865376 (+ 8016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:14.871240 (+ 5864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.871245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.871250 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.871572 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.871997 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.872002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.872004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.872117 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.872699 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:14.872707 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:14.872836 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 10:42:14.872950 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 10:42:14.873268 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 10:42:14.874310 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.852007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.2.8.60:37069)
0924 10:42:14.852495 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:14.852507 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:14.852870 (+ 363us) server_negotiation.cc:407] Connection header received
0924 10:42:14.853035 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:14.853039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:14.853099 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:14.853776 (+ 677us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:14.853879 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.854738 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.855958 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.856351 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.856831 (+ 480us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.865540 (+ 8709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:14.865545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:14.865548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:14.865578 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:14.871086 (+ 5508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.871739 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.871746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.871751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.871844 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.872282 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.872288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.872293 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.872542 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:14.873784 (+ 1242us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:14.873874 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 10:42:14.874036 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":224,"threads_started":1}
I20260924 10:42:14.874626 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.848595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:37069 (local address 127.0.0.1:45968)
0924 10:42:14.848711 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.848735 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:14.848802 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:14.849068 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:14.849070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:14.849227 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:14.849300 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.849304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.850218 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.850222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.850930 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.850940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.851773 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.851793 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.851898 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.852467 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:14.852481 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:14.866643 (+ 14162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:14.872185 (+ 5542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.872188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.872190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.872473 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.873505 (+ 1032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.873508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.873511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.873608 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.874116 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:14.874119 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:14.874230 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 10:42:14.874318 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 10:42:14.874473 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 10:42:14.875262 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.848687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.2.8.60:37069)
0924 10:42:14.848805 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:14.848810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:14.848824 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:42:14.848859 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:14.848864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:14.848905 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:14.849015 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:14.849412 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.850137 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.851058 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.851632 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.852098 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.866799 (+ 14701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:14.866802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:14.866805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:14.866828 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:14.872049 (+ 5221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.873290 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.873295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.873298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.873382 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.873737 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.873739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.873741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.874001 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:14.874903 (+ 902us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:14.874995 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 10:42:14.875116 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:42:14.877674 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.866655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:37069 (local address 127.0.0.1:45992)
0924 10:42:14.866765 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:14.866781 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:14.866865 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:14.867335 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:14.867338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:14.867545 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:14.867670 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.867679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.868968 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.869002 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.869938 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:14.869948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.870834 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.870851 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:14.870959 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.871513 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:14.871535 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:14.873503 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:14.875944 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.875952 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.875959 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.876222 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.876557 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:14.876560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:14.876563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:14.876655 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:14.877090 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:14.877094 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:14.877195 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 10:42:14.877328 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 10:42:14.877477 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 10:42:14.877741 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.866798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45992 (local address 127.2.8.60:37069)
0924 10:42:14.867120 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:14.867128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:14.867141 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:42:14.867171 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:14.867173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:14.867208 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:14.867328 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:14.867800 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.868844 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.870101 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.870560 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.870694 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.873669 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:14.873674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:14.873677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:14.873704 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:14.875811 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.876337 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.876344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.876348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.876441 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.876759 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.876766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.876768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.876950 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:14.877104 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:14.877374 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 10:42:14.877551 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
I20260924 10:42:14.878026 15531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5: Bootstrap starting.
I20260924 10:42:14.878038 15456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc: Bootstrap starting.
I20260924 10:42:14.880884 15456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:14.880884 15531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:14.881877 15531 log.cc:824] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:14.881923 15456 log.cc:824] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:14.883541 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.854288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55526 (local address 127.2.8.61:35817)
0924 10:42:14.854441 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:14.854446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:14.854631 (+ 185us) server_negotiation.cc:407] Connection header received
0924 10:42:14.854909 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:14.854912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:14.854972 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:14.855057 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:14.856180 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.857074 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.858140 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.858463 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.858554 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.867690 (+ 9136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:14.867694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:14.867696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:14.867720 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:14.873995 (+ 6275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.874966 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.874969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.874971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.875048 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.882084 (+ 7036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.882088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.882090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.882361 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:14.882582 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:14.883259 (+ 677us) server_negotiation.cc:299] Negotiation successful
0924 10:42:14.883381 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 10:42:14.884032 15456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc: No bootstrap required, opened a new log
I20260924 10:42:14.884032 15531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5: No bootstrap required, opened a new log
I20260924 10:42:14.884258 15608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80: Bootstrap starting.
I20260924 10:42:14.886663 15608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:14.887497 15608 log.cc:824] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:14.888115 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:42:14.888309 15531 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } }
I20260924 10:42:14.888309 15456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } }
I20260924 10:42:14.888537 15456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:14.888537 15531 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:14.888594 15456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 144b387f7b214adab97adab6c0982acc, State: Initialized, Role: FOLLOWER
I20260924 10:42:14.888594 15531 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cdef7d694e43a482a547994e63c5f5, State: Initialized, Role: FOLLOWER
I20260924 10:42:14.889081 15456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } }
I20260924 10:42:14.889081 15531 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } }
I20260924 10:42:14.890082 15608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80: No bootstrap required, opened a new log
I20260924 10:42:14.890357 15531 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:42:14.890357 15456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:42:14.894472 15608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } }
I20260924 10:42:14.894658 15620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } } }
I20260924 10:42:14.894706 15608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:14.894758 15620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:42:14.894766 15608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fff1de8219b4bb4ad5b6f8e63c5ae80, State: Initialized, Role: FOLLOWER
I20260924 10:42:14.895241 15608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } }
W20260924 10:42:14.895884 15645 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:42:14.895958 15645 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:42:14.896696 15608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:42:14.899153 15648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } } }
I20260924 10:42:14.899259 15648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 10:42:14.901408 15661 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:42:14.901484 15661 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:42:14.892669 15619 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } } }
I20260924 10:42:14.901633 15619 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:42:14.901757 15388 master_runner.cc:263] No leader master found from master 144b387f7b214adab97adab6c0982acc
I20260924 10:42:14.901825 15388 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:42:14.902370 15463 master_runner.cc:263] No leader master found from master c2cdef7d694e43a482a547994e63c5f5
I20260924 10:42:14.902436 15463 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 10:42:14.902601 15644 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:42:14.902678 15644 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:42:14.904052 15538 master_runner.cc:263] No leader master found from master 8fff1de8219b4bb4ad5b6f8e63c5ae80
I20260924 10:42:14.904117 15538 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:42:14.906265 15648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:42:14.906443 15648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } }
I20260924 10:42:14.907385 15648 leader_election.cc:288] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 144b387f7b214adab97adab6c0982acc (127.2.8.62:45925), c2cdef7d694e43a482a547994e63c5f5 (127.2.8.61:35817)
I20260924 10:42:14.907900 15505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2cdef7d694e43a482a547994e63c5f5" is_pre_election: true
I20260924 10:42:14.908210 15505 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fff1de8219b4bb4ad5b6f8e63c5ae80 in term 0.
I20260924 10:42:14.908344 15430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "144b387f7b214adab97adab6c0982acc" is_pre_election: true
I20260924 10:42:14.908598 15430 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fff1de8219b4bb4ad5b6f8e63c5ae80 in term 0.
I20260924 10:42:14.908646 15556 leader_election.cc:302] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fff1de8219b4bb4ad5b6f8e63c5ae80, c2cdef7d694e43a482a547994e63c5f5; no voters:
I20260924 10:42:14.909230 15648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:42:14.909328 15648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:42:14.909389 15648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:14.910935 15648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } }
I20260924 10:42:14.911556 15430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "144b387f7b214adab97adab6c0982acc"
I20260924 10:42:14.911672 15430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:14.913139 15430 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fff1de8219b4bb4ad5b6f8e63c5ae80 in term 1.
I20260924 10:42:14.913451 15556 leader_election.cc:302] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 144b387f7b214adab97adab6c0982acc, 8fff1de8219b4bb4ad5b6f8e63c5ae80; no voters:
I20260924 10:42:14.914057 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.895706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.2.8.61:35817)
0924 10:42:14.895811 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:14.895815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:14.903301 (+ 7486us) server_negotiation.cc:407] Connection header received
0924 10:42:14.903964 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:14.903967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:14.904018 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:14.904090 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:14.904928 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.905495 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.906872 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.907215 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.907301 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.909911 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:14.909917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:14.909921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:14.909946 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:14.911725 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.912338 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.912344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.912348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.912434 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.912780 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.912783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.912785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.913052 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:14.913186 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:14.913718 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 10:42:14.913865 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 10:42:14.914796 15648 leader_election.cc:288] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [CANDIDATE]: Term 1 election: Requested vote from peers 144b387f7b214adab97adab6c0982acc (127.2.8.62:45925), c2cdef7d694e43a482a547994e63c5f5 (127.2.8.61:35817)
I20260924 10:42:14.914985 15505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2cdef7d694e43a482a547994e63c5f5"
I20260924 10:42:14.915104 15505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:14.916512 15505 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fff1de8219b4bb4ad5b6f8e63c5ae80 in term 1.
I20260924 10:42:14.914906 15648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:42:14.918090 15648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [term 1 LEADER]: Becoming Leader. State: Replica: 8fff1de8219b4bb4ad5b6f8e63c5ae80, State: Running, Role: LEADER
I20260924 10:42:14.918437 15648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } }
I20260924 10:42:14.919844 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.895102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.2.8.62:45925)
0924 10:42:14.895203 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:14.895207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:14.901029 (+ 5822us) server_negotiation.cc:407] Connection header received
0924 10:42:14.902005 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:14.902009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:14.902057 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:14.902109 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:14.903082 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.903744 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.905056 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.905413 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.905524 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.915688 (+ 10164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:14.915692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:14.915695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:14.915722 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:14.917683 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.918325 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.918329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.918331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.918422 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.918849 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.918852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.918854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.919079 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:14.919197 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:14.919551 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 10:42:14.919691 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 10:42:14.921528 15664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fff1de8219b4bb4ad5b6f8e63c5ae80. Latest consensus state: current_term: 1 leader_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } } }
I20260924 10:42:14.921675 15664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:14.922112 15666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:42:14.923175 15666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:42:14.928603 15505 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5 [term 1 FOLLOWER]: Refusing update from remote peer 8fff1de8219b4bb4ad5b6f8e63c5ae80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:42:14.929363 15664 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:42:14.931248 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:14.896184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.2.8.60:37069)
0924 10:42:14.896282 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:14.896288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:14.913234 (+ 16946us) server_negotiation.cc:407] Connection header received
0924 10:42:14.913406 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:14.913409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:14.913455 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:14.913502 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:14.914292 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.915083 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.925090 (+ 10007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:14.925471 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:14.925546 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:14.927249 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:14.927253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:14.927256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:14.927281 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:14.929468 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.929941 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.929944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.929946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.930021 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:14.930324 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:14.930328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:14.930330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:14.930524 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:14.930624 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:14.930970 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 10:42:14.931100 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 10:42:14.933477 15619 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fff1de8219b4bb4ad5b6f8e63c5ae80. Latest consensus state: current_term: 1 leader_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } } }
I20260924 10:42:14.933609 15619 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:42:14.935547 15430 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc [term 1 FOLLOWER]: Refusing update from remote peer 8fff1de8219b4bb4ad5b6f8e63c5ae80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:42:14.936203 15664 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:42:14.940428 15620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fff1de8219b4bb4ad5b6f8e63c5ae80. Latest consensus state: current_term: 1 leader_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } } }
I20260924 10:42:14.940560 15620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:42:14.950656 15664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } } }
I20260924 10:42:14.950788 15664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:14.951776 15620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } } }
I20260924 10:42:14.951895 15620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:42:14.952131 15667 mvcc.cc:205] Tried to move back new op lower bound from 7332849807059845120 to 7332849807030747136. Current Snapshot: MvccSnapshot[applied={T|T < 7332849807059845120}]
I20260924 10:42:14.953066 15648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } } }
I20260924 10:42:14.953197 15648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80 [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:14.954105 15619 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144b387f7b214adab97adab6c0982acc" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45925 } } peers { permanent_uuid: "c2cdef7d694e43a482a547994e63c5f5" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 35817 } } peers { permanent_uuid: "8fff1de8219b4bb4ad5b6f8e63c5ae80" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 37069 } } }
I20260924 10:42:14.954232 15619 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:42:14.959903 15670 mvcc.cc:205] Tried to move back new op lower bound from 7332849807059845120 to 7332849807030747136. Current Snapshot: MvccSnapshot[applied={T|T < 7332849807059845120}]
I20260924 10:42:14.960430 15668 mvcc.cc:205] Tried to move back new op lower bound from 7332849807059845120 to 7332849807030747136. Current Snapshot: MvccSnapshot[applied={T|T < 7332849807059845120}]
I20260924 10:42:14.962451 15666 catalog_manager.cc:1347] Generated new cluster ID: 54c6eabcd8ec40c5a1ae0abc4f69f408
I20260924 10:42:14.962527 15666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:42:15.023247 15666 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:42:15.024740 15666 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:42:15.048048 15666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fff1de8219b4bb4ad5b6f8e63c5ae80: Generated new TSK 0
I20260924 10:42:15.049091 15666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:42:15.897509 15645 catalog_manager.cc:1259] Loaded cluster ID: 54c6eabcd8ec40c5a1ae0abc4f69f408
I20260924 10:42:15.897612 15645 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5: loading cluster ID for follower catalog manager: success
I20260924 10:42:15.899560 15645 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5: acquiring CA information for follower catalog manager: success
I20260924 10:42:15.900782 15645 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2cdef7d694e43a482a547994e63c5f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:42:15.903654 15644 catalog_manager.cc:1259] Loaded cluster ID: 54c6eabcd8ec40c5a1ae0abc4f69f408
I20260924 10:42:15.903721 15644 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc: loading cluster ID for follower catalog manager: success
I20260924 10:42:15.905205 15644 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc: acquiring CA information for follower catalog manager: success
I20260924 10:42:15.906246 15644 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 144b387f7b214adab97adab6c0982acc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:42:16.124186 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 15388
I20260924 10:42:16.130431 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 15463
I20260924 10:42:16.136000 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 15538
2026-09-24T10:42:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3106 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:42:16 dist-test-slave-j38h krb5kdc[15682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:42:16 dist-test-slave-j38h krb5kdc[15682](info): set up 2 sockets
Sep 24 10:42:16 dist-test-slave-j38h krb5kdc[15682](info): commencing operation
krb5kdc: starting...
W20260924 10:42:17.225190 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:42:17 dist-test-slave-j38h krb5kdc[15682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:42:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:42:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:42:17.350231 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:42405
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:44025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:42405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:17.458936 15695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:17.459219 15695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:17.459260 15695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:17.463258 15695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:42:17.463315 15695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:17.463335 15695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:17.463351 15695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:42:17.463366 15695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:42:17.468505 15695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:44025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:42405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:42405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:17.469602 15695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:17.470577 15695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:17.478776 15701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:17.478848 15695 server_base.cc:1034] running on GCE node
W20260924 10:42:17.478781 15700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:17.478827 15703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:17.479629 15695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:17.480676 15695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:17.481830 15695 hybrid_clock.cc:648] HybridClock initialized: now 1790246537481790 us; error 54 us; skew 500 ppm
Sep 24 10:42:17 dist-test-slave-j38h krb5kdc[15682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246537, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:17.484678 15695 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:42:17.485630 15695 webserver.cc:460] Webserver started at http://127.2.8.62:44403/ using document root <none> and password file <none>
I20260924 10:42:17.486202 15695 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:17.486263 15695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:17.486479 15695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:42:17.488466 15695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "7fa4cd626ade4c919f27e10ef1514f65"
format_stamp: "Formatted at 2026-09-24 10:42:17 on dist-test-slave-j38h"
server_key: "6a5acb536d2062866f64a678c583c4de"
server_key_iv: "5b698b481689ea5522d80820c8418fed"
server_key_version: "encryptionkey@0"
I20260924 10:42:17.489094 15695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "7fa4cd626ade4c919f27e10ef1514f65"
format_stamp: "Formatted at 2026-09-24 10:42:17 on dist-test-slave-j38h"
server_key: "6a5acb536d2062866f64a678c583c4de"
server_key_iv: "5b698b481689ea5522d80820c8418fed"
server_key_version: "encryptionkey@0"
I20260924 10:42:17.492755 15695 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:42:17.495127 15710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:17.496320 15695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:42:17.496462 15695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "7fa4cd626ade4c919f27e10ef1514f65"
format_stamp: "Formatted at 2026-09-24 10:42:17 on dist-test-slave-j38h"
server_key: "6a5acb536d2062866f64a678c583c4de"
server_key_iv: "5b698b481689ea5522d80820c8418fed"
server_key_version: "encryptionkey@0"
I20260924 10:42:17.496579 15695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:17.509357 15695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:17.512521 15695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:17.512732 15695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:17.519542 15695 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:42405
I20260924 10:42:17.519553 15762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:42405 every 8 connection(s)
I20260924 10:42:17.520521 15695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:42:17.523275 15763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:17.526228 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 15695
I20260924 10:42:17.526364 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:42:17.526635 2080 external_mini_cluster.cc:1428] Setting key 4070e179470a48ac454e8c52efa9eef4
I20260924 10:42:17.528926 15763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65: Bootstrap starting.
I20260924 10:42:17.531741 15763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:17.532729 15763 log.cc:824] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:17.535171 15763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65: No bootstrap required, opened a new log
Sep 24 10:42:17 dist-test-slave-j38h krb5kdc[15682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:17.538573 15763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa4cd626ade4c919f27e10ef1514f65" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42405 } }
I20260924 10:42:17.538810 15763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:17.538875 15763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa4cd626ade4c919f27e10ef1514f65, State: Initialized, Role: FOLLOWER
I20260924 10:42:17.539354 15763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa4cd626ade4c919f27e10ef1514f65" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42405 } }
I20260924 10:42:17.539541 15763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:42:17.539640 15763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:42:17.539777 15763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:17.540747 15763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa4cd626ade4c919f27e10ef1514f65" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42405 } }
I20260924 10:42:17.541180 15763 leader_election.cc:302] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fa4cd626ade4c919f27e10ef1514f65; no voters:
I20260924 10:42:17.541579 15763 leader_election.cc:288] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:42:17.541671 15768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:42:17.541916 15768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa4cd626ade4c919f27e10ef1514f65, State: Running, Role: LEADER
I20260924 10:42:17.542281 15768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa4cd626ade4c919f27e10ef1514f65" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42405 } }
I20260924 10:42:17.542619 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:17.528035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.2.8.62:42405)
0924 10:42:17.528426 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:17.528450 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:17.528484 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:42:17.529173 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:17.529191 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:17.529515 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:17.529784 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:17.530891 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:17.531918 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:17.533226 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:17.533823 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:17.533942 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:17.536823 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:17.536828 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:17.536841 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:17.536866 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:17.539346 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:17.539841 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:17.539847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:17.539853 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:17.540010 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:17.540330 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:17.540335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:17.540337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:17.540750 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:17.540967 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:17.541268 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 10:42:17.541614 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":130,"threads_started":1}
I20260924 10:42:17.543066 15763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:42:17.544744 15770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fa4cd626ade4c919f27e10ef1514f65. Latest consensus state: current_term: 1 leader_uuid: "7fa4cd626ade4c919f27e10ef1514f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa4cd626ade4c919f27e10ef1514f65" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42405 } } }
I20260924 10:42:17.544869 15770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:17.545225 15769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fa4cd626ade4c919f27e10ef1514f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fa4cd626ade4c919f27e10ef1514f65" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42405 } } }
I20260924 10:42:17.545305 15769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65 [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:17.546123 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 10:42:17.548300 15783 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:42:17.548394 15783 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:42:17.548645 15780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:42:17.549459 15780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:42:17.555192 15780 catalog_manager.cc:1347] Generated new cluster ID: dbd01d43f8c0464ba628ae3dc67c9c8f
I20260924 10:42:17.555270 15780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:42:17.580525 15780 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:42:17.581692 15780 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:42:17.586922 15780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fa4cd626ade4c919f27e10ef1514f65: Generated new TSK 0
I20260924 10:42:17.587625 15780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:42:17.602418 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 15695
2026-09-24T10:42:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1464 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:42:17 dist-test-slave-j38h krb5kdc[15790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:42:17 dist-test-slave-j38h krb5kdc[15790](info): set up 2 sockets
Sep 24 10:42:17 dist-test-slave-j38h krb5kdc[15790](info): commencing operation
krb5kdc: starting...
W20260924 10:42:18.665400 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:42:18 dist-test-slave-j38h krb5kdc[15790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:42:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:42:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:42:18.786146 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:46121
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:46121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:18.898528 15803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:18.898823 15803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:18.898900 15803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:18.902925 15803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:42:18.903009 15803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:18.903046 15803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:18.903110 15803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:42:18.903151 15803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:42:18.908270 15803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:46121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:46121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:18.909385 15803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:18.910374 15803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:18.918774 15808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:18.918845 15803 server_base.cc:1034] running on GCE node
W20260924 10:42:18.918864 15809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:18.918776 15811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:18.919441 15803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:18.920499 15803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:18.921666 15803 hybrid_clock.cc:648] HybridClock initialized: now 1790246538921641 us; error 39 us; skew 500 ppm
Sep 24 10:42:18 dist-test-slave-j38h krb5kdc[15790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246538, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:18.936579 15803 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:42:18.937656 15803 webserver.cc:460] Webserver started at http://127.2.8.62:37245/ using document root <none> and password file <none>
I20260924 10:42:18.938247 15803 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:18.938310 15803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:18.938544 15803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:42:18.940649 15803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "c05fa2ac1c2745419924fd67f50e0c49"
format_stamp: "Formatted at 2026-09-24 10:42:18 on dist-test-slave-j38h"
server_key: "991de6c99f1bc4ef3ca1160aad9348f1"
server_key_iv: "3da720042aedc76285cbc9f7ee8d02d6"
server_key_version: "encryptionkey@0"
I20260924 10:42:18.941241 15803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "c05fa2ac1c2745419924fd67f50e0c49"
format_stamp: "Formatted at 2026-09-24 10:42:18 on dist-test-slave-j38h"
server_key: "991de6c99f1bc4ef3ca1160aad9348f1"
server_key_iv: "3da720042aedc76285cbc9f7ee8d02d6"
server_key_version: "encryptionkey@0"
I20260924 10:42:18.944864 15803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:42:18.947363 15818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:18.948566 15803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:42:18.948709 15803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "c05fa2ac1c2745419924fd67f50e0c49"
format_stamp: "Formatted at 2026-09-24 10:42:18 on dist-test-slave-j38h"
server_key: "991de6c99f1bc4ef3ca1160aad9348f1"
server_key_iv: "3da720042aedc76285cbc9f7ee8d02d6"
server_key_version: "encryptionkey@0"
I20260924 10:42:18.948820 15803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:18.960184 15803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:18.963438 15803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:18.963640 15803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:18.970465 15803 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:46121
I20260924 10:42:18.970747 15870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:46121 every 8 connection(s)
I20260924 10:42:18.971431 15803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:42:18.972209 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 15803
I20260924 10:42:18.972347 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:42:18.972656 2080 external_mini_cluster.cc:1428] Setting key b337cce3b531eec5168b3c2087b962db
I20260924 10:42:18.974807 15871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:18.981204 15871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49: Bootstrap starting.
Sep 24 10:42:18 dist-test-slave-j38h krb5kdc[15790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:18.983681 15871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:18.984385 15871 log.cc:824] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:18.986331 15871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49: No bootstrap required, opened a new log
I20260924 10:42:18.987345 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:18.973988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.2.8.62:46121)
0924 10:42:18.974371 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:18.974390 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:18.974430 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:42:18.975273 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:18.975295 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:18.975692 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:18.976049 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:18.976708 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:18.978227 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:18.979142 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:18.979836 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:18.979988 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:18.982795 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:18.982800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:18.982813 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:18.982838 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:18.984552 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:18.985090 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:18.985098 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:18.985107 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:18.985219 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:18.985521 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:18.985526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:18.985527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:18.985905 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:18.986103 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:18.986380 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 10:42:18.986668 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":99,"threads_started":1}
I20260924 10:42:18.989890 15871 raft_consensus.cc:348] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } }
I20260924 10:42:18.990124 15871 raft_consensus.cc:374] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:18.990206 15871 raft_consensus.cc:729] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05fa2ac1c2745419924fd67f50e0c49, State: Initialized, Role: FOLLOWER
I20260924 10:42:18.990659 15871 consensus_queue.cc:260] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } }
I20260924 10:42:18.990808 15871 raft_consensus.cc:388] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:42:18.990893 15871 raft_consensus.cc:482] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:42:18.990988 15871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:18.991883 15871 raft_consensus.cc:504] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } }
I20260924 10:42:18.992218 15871 leader_election.cc:302] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c05fa2ac1c2745419924fd67f50e0c49; no voters:
I20260924 10:42:18.992511 15871 leader_election.cc:288] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:42:18.992664 15876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:42:18.992895 15876 raft_consensus.cc:686] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 1 LEADER]: Becoming Leader. State: Replica: c05fa2ac1c2745419924fd67f50e0c49, State: Running, Role: LEADER
I20260924 10:42:18.993273 15876 consensus_queue.cc:237] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } }
I20260924 10:42:18.993781 15871 sys_catalog.cc:564] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:42:18.994875 15878 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c05fa2ac1c2745419924fd67f50e0c49. Latest consensus state: current_term: 1 leader_uuid: "c05fa2ac1c2745419924fd67f50e0c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } } }
I20260924 10:42:18.994995 15878 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:18.994966 15877 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c05fa2ac1c2745419924fd67f50e0c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } } }
I20260924 10:42:18.995064 15877 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:18.995350 15885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:42:18.998371 15885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:42:19.003551 15885 catalog_manager.cc:1347] Generated new cluster ID: fadab5abd7174a50aad7e64f795cd7be
I20260924 10:42:19.003618 15885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:42:19.023909 15885 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:42:19.024785 15885 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:42:19.030406 15885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49: Generated new TSK 0
I20260924 10:42:19.031213 15885 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:42:19.071893 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:46121
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:19.181134 15897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:19.181393 15897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:19.181437 15897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:19.185398 15897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:19.185454 15897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:19.185567 15897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:42:19.190763 15897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:19.191820 15897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:19.192891 15897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:19.202916 15903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:19.202955 15902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:19.202916 15905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:19.203507 15897 server_base.cc:1034] running on GCE node
I20260924 10:42:19.203935 15897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:19.204560 15897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:19.205770 15897 hybrid_clock.cc:648] HybridClock initialized: now 1790246539205749 us; error 29 us; skew 500 ppm
Sep 24 10:42:19 dist-test-slave-j38h krb5kdc[15790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:19.208690 15897 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:42:19.209615 15897 webserver.cc:460] Webserver started at http://127.2.8.1:35313/ using document root <none> and password file <none>
I20260924 10:42:19.210191 15897 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:19.210284 15897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:19.210515 15897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:42:19.212534 15897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "4748cb5a7f29474b8f48929756786b22"
format_stamp: "Formatted at 2026-09-24 10:42:19 on dist-test-slave-j38h"
server_key: "c880b5494e43068b180e29f8c09b0970"
server_key_iv: "fdb52bf81d8577135a57b62519c39edc"
server_key_version: "encryptionkey@0"
I20260924 10:42:19.213168 15897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "4748cb5a7f29474b8f48929756786b22"
format_stamp: "Formatted at 2026-09-24 10:42:19 on dist-test-slave-j38h"
server_key: "c880b5494e43068b180e29f8c09b0970"
server_key_iv: "fdb52bf81d8577135a57b62519c39edc"
server_key_version: "encryptionkey@0"
I20260924 10:42:19.216810 15897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:42:19.219219 15912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:19.220397 15897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:42:19.220527 15897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "4748cb5a7f29474b8f48929756786b22"
format_stamp: "Formatted at 2026-09-24 10:42:19 on dist-test-slave-j38h"
server_key: "c880b5494e43068b180e29f8c09b0970"
server_key_iv: "fdb52bf81d8577135a57b62519c39edc"
server_key_version: "encryptionkey@0"
I20260924 10:42:19.220633 15897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:19.241875 15897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:19.245327 15897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:19.245524 15897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:19.246112 15897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:19.246904 15897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:42:19.246974 15897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:19.247046 15897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:19.247097 15897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:19.256219 15897 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:33271
I20260924 10:42:19.256242 16025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:33271 every 8 connection(s)
I20260924 10:42:19.257165 15897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:42:19.258406 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 15897
I20260924 10:42:19.258543 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:42:19.258882 2080 external_mini_cluster.cc:1428] Setting key e2aa9f6364692ca1322403d2eab1235a
Sep 24 10:42:19 dist-test-slave-j38h krb5kdc[15790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
I20260924 10:42:19.272568 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.259173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:58577 (local address 127.2.8.62:46121)
0924 10:42:19.259337 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:19.259342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:19.260346 (+ 1004us) server_negotiation.cc:407] Connection header received
0924 10:42:19.261792 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:19.261795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:19.261850 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:19.261928 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:19.263170 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.263715 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.265613 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.265848 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.265936 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.268841 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:19.268845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:19.268848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:19.268877 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:19.270388 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:19.270978 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:19.270981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:19.270984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:19.271064 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:19.271431 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:19.271434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:19.271436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:19.271669 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:19.271777 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:19.272268 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 10:42:19.272397 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:42:19.273725 16028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.259618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:58577)
0924 10:42:19.260153 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:19.260210 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:19.261575 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:19.262116 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:19.262123 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:19.262480 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:19.262989 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:19.263012 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.263850 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.263857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:19.265483 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:19.265498 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.265958 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.266026 (+ 68us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:19.266284 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.266972 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:19.266992 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:19.268636 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:19.270507 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:19.270513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:19.270527 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:19.270862 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:19.271163 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:19.271167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:19.271169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:19.271326 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:19.271772 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:19.271778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:19.272026 (+ 248us) client_negotiation.cc:769] Sending connection context
0924 10:42:19.272229 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 10:42:19.272553 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":126,"threads_started":1}
I20260924 10:42:19.275130 16026 heartbeater.cc:347] Connected to a master server at 127.2.8.62:46121
I20260924 10:42:19.275382 16026 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:19.275897 16026 heartbeater.cc:511] Master 127.2.8.62:46121 requested a full tablet report, sending...
I20260924 10:42:19.277459 15835 ts_manager.cc:194] Registered new tserver with Master: 4748cb5a7f29474b8f48929756786b22 (127.2.8.1:33271)
I20260924 10:42:19.278584 15835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:58577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:42:19.288635 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:46121
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:19.398120 16031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:19.398377 16031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:19.398418 16031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:19.402325 16031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:19.402374 16031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:19.402431 16031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:42:19.408167 16031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:19.409157 16031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:19.410102 16031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:19.419296 16036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:19.419308 16039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:19.419324 16037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:19.419456 16031 server_base.cc:1034] running on GCE node
I20260924 10:42:19.419966 16031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:19.420589 16031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:19.421766 16031 hybrid_clock.cc:648] HybridClock initialized: now 1790246539421735 us; error 43 us; skew 500 ppm
Sep 24 10:42:19 dist-test-slave-j38h krb5kdc[15790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:19.424528 16031 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:42:19.425451 16031 webserver.cc:460] Webserver started at http://127.2.8.2:46211/ using document root <none> and password file <none>
I20260924 10:42:19.426002 16031 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:19.426101 16031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:19.426332 16031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:42:19.428318 16031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "999bc3d14dc64566b08e1dd16f85f7c0"
format_stamp: "Formatted at 2026-09-24 10:42:19 on dist-test-slave-j38h"
server_key: "0366a5916366bd9d90622c27766b4a26"
server_key_iv: "1b04e1c8a6e45863632131e40e294daf"
server_key_version: "encryptionkey@0"
I20260924 10:42:19.428938 16031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "999bc3d14dc64566b08e1dd16f85f7c0"
format_stamp: "Formatted at 2026-09-24 10:42:19 on dist-test-slave-j38h"
server_key: "0366a5916366bd9d90622c27766b4a26"
server_key_iv: "1b04e1c8a6e45863632131e40e294daf"
server_key_version: "encryptionkey@0"
I20260924 10:42:19.432741 16031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 10:42:19.435142 16046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:19.436290 16031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:42:19.436421 16031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "999bc3d14dc64566b08e1dd16f85f7c0"
format_stamp: "Formatted at 2026-09-24 10:42:19 on dist-test-slave-j38h"
server_key: "0366a5916366bd9d90622c27766b4a26"
server_key_iv: "1b04e1c8a6e45863632131e40e294daf"
server_key_version: "encryptionkey@0"
I20260924 10:42:19.436532 16031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:19.461375 16031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:19.464251 16031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:19.464465 16031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:19.465103 16031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:19.465950 16031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:42:19.466022 16031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:19.466133 16031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:19.466188 16031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:19.474931 16031 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:41243
I20260924 10:42:19.474964 16159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:41243 every 8 connection(s)
I20260924 10:42:19.475821 16031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:42:19.485036 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 16031
I20260924 10:42:19.485184 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
Sep 24 10:42:19 dist-test-slave-j38h krb5kdc[15790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:19.485548 2080 external_mini_cluster.cc:1428] Setting key 294c8fbb494c97b7ba48060d5c41600c
I20260924 10:42:19.489740 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.477723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:60597 (local address 127.2.8.62:46121)
0924 10:42:19.477859 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:19.477863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:19.478741 (+ 878us) server_negotiation.cc:407] Connection header received
0924 10:42:19.480014 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:19.480017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:19.480067 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:19.480150 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:19.481429 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.481994 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.483135 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.483368 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.483453 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.486120 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:19.486124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:19.486126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:19.486150 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:19.487663 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:19.488207 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:19.488211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:19.488213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:19.488294 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:19.488643 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:19.488648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:19.488650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:19.488827 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:19.488937 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:19.489459 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 10:42:19.489583 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:42:19.490563 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.478096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:60597)
0924 10:42:19.478589 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:19.478631 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:19.479788 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:19.480336 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:19.480343 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:19.480717 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:19.481289 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:19.481304 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.482136 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.482139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:19.483019 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:19.483029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.483458 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.483508 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:19.483648 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.484235 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:19.484251 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:19.485938 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:19.487806 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:19.487811 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:19.487834 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:19.488094 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:19.488394 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:19.488397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:19.488399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:19.488547 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:19.488937 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:19.488945 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:19.489218 (+ 273us) client_negotiation.cc:769] Sending connection context
0924 10:42:19.489406 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 10:42:19.489742 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":128,"threads_started":1}
I20260924 10:42:19.491905 16160 heartbeater.cc:347] Connected to a master server at 127.2.8.62:46121
I20260924 10:42:19.492159 16160 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:19.492641 16160 heartbeater.cc:511] Master 127.2.8.62:46121 requested a full tablet report, sending...
I20260924 10:42:19.493752 15835 ts_manager.cc:194] Registered new tserver with Master: 999bc3d14dc64566b08e1dd16f85f7c0 (127.2.8.2:41243)
I20260924 10:42:19.494515 15835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:60597
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:42:19.512454 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:46121
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:19.623579 16165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:19.623858 16165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:19.623899 16165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:19.627965 16165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:19.628023 16165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:19.628082 16165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:42:19.633168 16165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:19.634086 16165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:19.635010 16165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:19.644560 16170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:19.644665 16165 server_base.cc:1034] running on GCE node
W20260924 10:42:19.644569 16171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:19.644565 16173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:19.645216 16165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:19.645905 16165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:19.647087 16165 hybrid_clock.cc:648] HybridClock initialized: now 1790246539647078 us; error 31 us; skew 500 ppm
Sep 24 10:42:19 dist-test-slave-j38h krb5kdc[15790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:19.649909 16165 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:42:19.650780 16165 webserver.cc:460] Webserver started at http://127.2.8.3:35005/ using document root <none> and password file <none>
I20260924 10:42:19.651347 16165 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:19.651438 16165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:19.651698 16165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:42:19.653890 16165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "7c09872922854c8a82c4c5d6f11683b4"
format_stamp: "Formatted at 2026-09-24 10:42:19 on dist-test-slave-j38h"
server_key: "2db522ebd38ed1287cc9b1cacd2c87da"
server_key_iv: "30dd16340e4df5aa480e9ec5b806dcf0"
server_key_version: "encryptionkey@0"
I20260924 10:42:19.654448 16165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "7c09872922854c8a82c4c5d6f11683b4"
format_stamp: "Formatted at 2026-09-24 10:42:19 on dist-test-slave-j38h"
server_key: "2db522ebd38ed1287cc9b1cacd2c87da"
server_key_iv: "30dd16340e4df5aa480e9ec5b806dcf0"
server_key_version: "encryptionkey@0"
I20260924 10:42:19.658192 16165 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 10:42:19.660565 16180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:19.661867 16165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:42:19.662004 16165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "7c09872922854c8a82c4c5d6f11683b4"
format_stamp: "Formatted at 2026-09-24 10:42:19 on dist-test-slave-j38h"
server_key: "2db522ebd38ed1287cc9b1cacd2c87da"
server_key_iv: "30dd16340e4df5aa480e9ec5b806dcf0"
server_key_version: "encryptionkey@0"
I20260924 10:42:19.662117 16165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:19.703029 16165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:19.706189 16165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:19.706405 16165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:19.707006 16165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:19.707828 16165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:42:19.707897 16165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:19.707967 16165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:19.708022 16165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:19.717314 16165 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:46731
I20260924 10:42:19.717348 16293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:46731 every 8 connection(s)
I20260924 10:42:19.718191 16165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:42:19.718894 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 16165
I20260924 10:42:19.719036 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:42:19.719309 2080 external_mini_cluster.cc:1428] Setting key 079f08c1f9a4fb0256e39be0e706adf0
Sep 24 10:42:19 dist-test-slave-j38h krb5kdc[15790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246539, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:19.732110 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.720062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:58857 (local address 127.2.8.62:46121)
0924 10:42:19.720223 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:19.720227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:19.721641 (+ 1414us) server_negotiation.cc:407] Connection header received
0924 10:42:19.722893 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:19.722896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:19.722939 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:19.723015 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:19.724221 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.724759 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.725935 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.726145 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.726233 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.728667 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:19.728671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:19.728674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:19.728705 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:19.730065 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:19.730583 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:19.730588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:19.730590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:19.730671 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:19.731009 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:19.731012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:19.731014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:19.731201 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:19.731310 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:19.731822 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 10:42:19.731940 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:42:19.733089 16296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.720504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:58857)
0924 10:42:19.721520 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:19.721565 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:19.722685 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:19.723184 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:19.723191 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:19.723533 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:19.724087 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:19.724102 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.724898 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.724900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:19.725833 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:19.725842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.726236 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.726279 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:19.726409 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.727052 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:19.727068 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:19.728500 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:19.730204 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:19.730210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:19.730225 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:19.730478 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:19.730763 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:19.730767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:19.730769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:19.730906 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:19.731307 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:19.731314 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:19.731558 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 10:42:19.731779 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 10:42:19.732089 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":141,"threads_started":1}
I20260924 10:42:19.734153 16294 heartbeater.cc:347] Connected to a master server at 127.2.8.62:46121
I20260924 10:42:19.734419 16294 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:19.734910 16294 heartbeater.cc:511] Master 127.2.8.62:46121 requested a full tablet report, sending...
I20260924 10:42:19.735867 15835 ts_manager.cc:194] Registered new tserver with Master: 7c09872922854c8a82c4c5d6f11683b4 (127.2.8.3:46731)
I20260924 10:42:19.736424 15835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:58857
I20260924 10:42:19.745127 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:42:19.754160 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.747030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.2.8.62:46121)
0924 10:42:19.747154 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:19.747158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:19.747341 (+ 183us) server_negotiation.cc:407] Connection header received
0924 10:42:19.747430 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:19.747432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:19.747468 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:19.747532 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:19.748204 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.748727 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.749561 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.749790 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.749866 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.751130 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:19.751132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:19.751134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:19.751154 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:19.752636 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:19.753088 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:19.753091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:19.753092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:19.753198 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:19.753432 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:19.753435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:19.753436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:19.753660 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:19.753768 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:19.753958 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 10:42:19.754044 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:42:19.756856 15835 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:42:19.760095 15835 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:42:19.774681 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.768508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56200 (local address 127.2.8.1:33271)
0924 10:42:19.769493 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:19.769510 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:19.769534 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:42:19.769609 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:19.769837 (+ 228us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:19.770081 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:19.770253 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:19.770608 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.771830 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.773431 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.774002 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.774131 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.774294 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:19.774436 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 10:42:19.774539 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":132,"threads_started":1}
I20260924 10:42:19.774675 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.768355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:33271 (local address 127.0.0.1:56200)
0924 10:42:19.768696 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:19.768745 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:19.768872 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:19.770268 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:19.770273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:19.770295 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:19.770467 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:19.770477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.772011 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.772028 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:19.773285 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:19.773295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.774131 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.774135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:19.774252 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.774289 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 10:42:19.774405 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 10:42:19.774525 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":102,"thread_start_us":89,"threads_started":1}
I20260924 10:42:19.774922 16308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.769126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:46731 (local address 127.0.0.1:50874)
0924 10:42:19.769362 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:19.769374 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:19.769446 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:19.770362 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:19.770365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:19.770381 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:19.770483 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:19.770489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.771785 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.771789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:19.773285 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:19.773295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.774559 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.774563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:19.774646 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.774670 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 10:42:19.774714 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 10:42:19.774779 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":95,"thread_start_us":95,"threads_started":1}
I20260924 10:42:19.775530 16307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.768590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:41243 (local address 127.0.0.1:49902)
0924 10:42:19.769141 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:19.769154 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:19.769260 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:19.770550 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:19.770556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:19.770575 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:19.770710 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:19.770716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.772091 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.772094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:19.774096 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:19.774108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.775157 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.775160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:19.775252 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.775274 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:42:19.775332 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 10:42:19.775397 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":82,"thread_start_us":92,"threads_started":1}
I20260924 10:42:19.775568 16310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.769506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.2.8.3:46731)
0924 10:42:19.769841 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:19.769857 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:19.769953 (+ 96us) server_negotiation.cc:407] Connection header received
0924 10:42:19.770032 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:19.770036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:19.770213 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:19.770720 (+ 507us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:19.770887 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.771649 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.773415 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.774429 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.775127 (+ 698us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.775269 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:19.775320 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 10:42:19.775422 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":96,"threads_started":1}
I20260924 10:42:19.776254 16309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.769215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49902 (local address 127.2.8.2:41243)
0924 10:42:19.770015 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:19.770031 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:19.770053 (+ 22us) server_negotiation.cc:407] Connection header received
0924 10:42:19.770113 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:19.770117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:19.770330 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:19.770565 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:19.770833 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.771956 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.774245 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.775050 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.775148 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.775986 (+ 838us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:19.776028 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 10:42:19.776094 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":138,"threads_started":1}
I20260924 10:42:19.778420 15960 tablet_service.cc:1467] Processing CreateTablet for tablet f0193f84567242b381b600e71006fccc (DEFAULT_TABLE table=test-table [id=5c04ddde7011474a932ebb70259ccab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:42:19.778573 16228 tablet_service.cc:1467] Processing CreateTablet for tablet f0193f84567242b381b600e71006fccc (DEFAULT_TABLE table=test-table [id=5c04ddde7011474a932ebb70259ccab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:42:19.778594 16094 tablet_service.cc:1467] Processing CreateTablet for tablet f0193f84567242b381b600e71006fccc (DEFAULT_TABLE table=test-table [id=5c04ddde7011474a932ebb70259ccab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:42:19.779536 15960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0193f84567242b381b600e71006fccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:19.779547 16228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0193f84567242b381b600e71006fccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:19.779543 16094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0193f84567242b381b600e71006fccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:19.786067 16311 tablet_bootstrap.cc:492] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: Bootstrap starting.
I20260924 10:42:19.786983 16312 tablet_bootstrap.cc:492] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: Bootstrap starting.
I20260924 10:42:19.788365 16311 tablet_bootstrap.cc:654] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:19.788841 16312 tablet_bootstrap.cc:654] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:19.789350 16313 tablet_bootstrap.cc:492] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4: Bootstrap starting.
I20260924 10:42:19.789731 16312 log.cc:824] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:19.789799 16311 log.cc:824] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:19.791487 16313 tablet_bootstrap.cc:654] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:19.791790 16312 tablet_bootstrap.cc:492] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: No bootstrap required, opened a new log
I20260924 10:42:19.791971 16311 tablet_bootstrap.cc:492] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: No bootstrap required, opened a new log
I20260924 10:42:19.792062 16312 ts_tablet_manager.cc:1397] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:42:19.792183 16311 ts_tablet_manager.cc:1397] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 10:42:19.792754 16313 log.cc:824] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:19.794953 16313 tablet_bootstrap.cc:492] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4: No bootstrap required, opened a new log
I20260924 10:42:19.795132 16313 ts_tablet_manager.cc:1397] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:42:19.795979 16311 raft_consensus.cc:348] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } } peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } }
I20260924 10:42:19.796278 16311 raft_consensus.cc:374] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:19.796363 16311 raft_consensus.cc:729] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999bc3d14dc64566b08e1dd16f85f7c0, State: Initialized, Role: FOLLOWER
I20260924 10:42:19.796213 16312 raft_consensus.cc:348] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } } peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } }
I20260924 10:42:19.796835 16311 consensus_queue.cc:260] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } } peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } }
I20260924 10:42:19.796906 16312 raft_consensus.cc:374] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:19.797006 16312 raft_consensus.cc:729] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4748cb5a7f29474b8f48929756786b22, State: Initialized, Role: FOLLOWER
I20260924 10:42:19.797703 16160 heartbeater.cc:503] Master 127.2.8.62:46121 was elected leader, sending a full tablet report...
I20260924 10:42:19.797987 16311 ts_tablet_manager.cc:1428] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260924 10:42:19.798038 16312 consensus_queue.cc:260] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } } peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } }
I20260924 10:42:19.798662 16026 heartbeater.cc:503] Master 127.2.8.62:46121 was elected leader, sending a full tablet report...
I20260924 10:42:19.800051 16313 raft_consensus.cc:348] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } } peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } }
I20260924 10:42:19.800304 16313 raft_consensus.cc:374] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:19.800371 16313 raft_consensus.cc:729] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c09872922854c8a82c4c5d6f11683b4, State: Initialized, Role: FOLLOWER
I20260924 10:42:19.800901 16313 consensus_queue.cc:260] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } } peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } }
I20260924 10:42:19.801163 16312 ts_tablet_manager.cc:1428] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260924 10:42:19.801923 16313 ts_tablet_manager.cc:1428] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 10:42:19.802261 16294 heartbeater.cc:503] Master 127.2.8.62:46121 was elected leader, sending a full tablet report...
I20260924 10:42:19.889441 16319 raft_consensus.cc:482] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:42:19.889732 16319 raft_consensus.cc:504] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } } peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } }
I20260924 10:42:19.890877 16319 leader_election.cc:288] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4748cb5a7f29474b8f48929756786b22 (127.2.8.1:33271), 999bc3d14dc64566b08e1dd16f85f7c0 (127.2.8.2:41243)
I20260924 10:42:19.894626 16296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.891173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:41243 (local address 127.2.8.3:41689)
0924 10:42:19.891292 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:19.891311 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:19.891412 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:19.891789 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:19.891792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:19.891810 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:19.891925 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:19.891931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.892653 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.892659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:19.893502 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:19.893510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.894210 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.894213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:19.894324 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.894341 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:42:19.894422 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:42:19.894497 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 10:42:19.894629 16309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.891441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:41689 (local address 127.2.8.2:41243)
0924 10:42:19.891567 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:19.891570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:19.891582 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:42:19.891635 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:19.891637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:19.891676 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:19.891778 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:19.892039 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.892533 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.893627 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.894094 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.894209 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.894272 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:19.894449 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 10:42:19.894517 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:42:19.895381 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.891173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:33271 (local address 127.2.8.3:44075)
0924 10:42:19.891646 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:19.891657 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:19.891777 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:19.892071 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:19.892074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:19.892090 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:19.892228 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:19.892234 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.893081 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.893085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:19.894042 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:19.894055 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.895035 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.895039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:19.895144 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.895173 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 10:42:19.895237 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 10:42:19.895284 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":52,"thread_start_us":134,"threads_started":1}
I20260924 10:42:19.895341 16114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0193f84567242b381b600e71006fccc" candidate_uuid: "7c09872922854c8a82c4c5d6f11683b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" is_pre_election: true
I20260924 10:42:19.895463 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.891325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:44075 (local address 127.2.8.1:33271)
0924 10:42:19.891459 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:19.891464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:19.891755 (+ 291us) server_negotiation.cc:407] Connection header received
0924 10:42:19.891910 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:19.891918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:19.891966 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:19.892059 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:19.892354 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.892901 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.894174 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.894883 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.895019 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.895113 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:19.895258 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 10:42:19.895335 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:42:19.895742 16114 raft_consensus.cc:2454] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c09872922854c8a82c4c5d6f11683b4 in term 0.
I20260924 10:42:19.896067 15980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0193f84567242b381b600e71006fccc" candidate_uuid: "7c09872922854c8a82c4c5d6f11683b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4748cb5a7f29474b8f48929756786b22" is_pre_election: true
I20260924 10:42:19.896313 16183 leader_election.cc:302] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c09872922854c8a82c4c5d6f11683b4, 999bc3d14dc64566b08e1dd16f85f7c0; no voters:
I20260924 10:42:19.896368 15980 raft_consensus.cc:2454] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c09872922854c8a82c4c5d6f11683b4 in term 0.
I20260924 10:42:19.896622 16319 raft_consensus.cc:2781] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:42:19.896739 16319 raft_consensus.cc:482] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:42:19.896793 16319 raft_consensus.cc:3037] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:19.897828 16319 raft_consensus.cc:504] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } } peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } }
I20260924 10:42:19.898154 16319 leader_election.cc:288] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [CANDIDATE]: Term 1 election: Requested vote from peers 4748cb5a7f29474b8f48929756786b22 (127.2.8.1:33271), 999bc3d14dc64566b08e1dd16f85f7c0 (127.2.8.2:41243)
I20260924 10:42:19.898495 15980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0193f84567242b381b600e71006fccc" candidate_uuid: "7c09872922854c8a82c4c5d6f11683b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4748cb5a7f29474b8f48929756786b22"
I20260924 10:42:19.898569 16114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0193f84567242b381b600e71006fccc" candidate_uuid: "7c09872922854c8a82c4c5d6f11683b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999bc3d14dc64566b08e1dd16f85f7c0"
I20260924 10:42:19.898645 15980 raft_consensus.cc:3037] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:19.898720 16114 raft_consensus.cc:3037] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:19.899652 15980 raft_consensus.cc:2454] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c09872922854c8a82c4c5d6f11683b4 in term 1.
I20260924 10:42:19.899911 16114 raft_consensus.cc:2454] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c09872922854c8a82c4c5d6f11683b4 in term 1.
I20260924 10:42:19.899921 16184 leader_election.cc:302] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4748cb5a7f29474b8f48929756786b22, 7c09872922854c8a82c4c5d6f11683b4; no voters:
I20260924 10:42:19.900084 16319 raft_consensus.cc:2781] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:42:19.900321 16319 raft_consensus.cc:686] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [term 1 LEADER]: Becoming Leader. State: Replica: 7c09872922854c8a82c4c5d6f11683b4, State: Running, Role: LEADER
I20260924 10:42:19.900698 16319 consensus_queue.cc:237] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } } peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } }
I20260924 10:42:19.903692 15835 catalog_manager.cc:5495] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c09872922854c8a82c4c5d6f11683b4 (127.2.8.3). New cstate: current_term: 1 leader_uuid: "7c09872922854c8a82c4c5d6f11683b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } health_report { overall_health: UNKNOWN } } }
I20260924 10:42:19.925729 16310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.922406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.2.8.3:46731)
0924 10:42:19.922558 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:19.922561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:19.922575 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:42:19.922662 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:19.922664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:19.922719 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:19.922809 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:42:19.923005 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.923514 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.924390 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:19.924592 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:19.924673 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:19.924923 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:42:19.925380 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:42:19.925464 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:19.925552 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 10:42:19.925600 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:42:19.931320 16114 raft_consensus.cc:1261] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0 [term 1 FOLLOWER]: Refusing update from remote peer 7c09872922854c8a82c4c5d6f11683b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:42:19.931329 15980 raft_consensus.cc:1261] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Refusing update from remote peer 7c09872922854c8a82c4c5d6f11683b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:42:19.932024 16319 consensus_queue.cc:1035] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:42:19.932243 16321 consensus_queue.cc:1035] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:42:19.941198 16325 mvcc.cc:205] Tried to move back new op lower bound from 7332849827551748096 to 7332849827435978752. Current Snapshot: MvccSnapshot[applied={T|T < 7332849827551748096}]
I20260924 10:42:19.947507 15835 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:42:19.947697 15835 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:42:19.950078 15835 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49: Sending DeleteTablet for 3 replicas of tablet f0193f84567242b381b600e71006fccc
I20260924 10:42:19.950888 16094 tablet_service.cc:1514] Processing DeleteTablet for tablet f0193f84567242b381b600e71006fccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:42:19 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:49902
I20260924 10:42:19.950941 15960 tablet_service.cc:1514] Processing DeleteTablet for tablet f0193f84567242b381b600e71006fccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:42:19 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:56200
I20260924 10:42:19.950989 16228 tablet_service.cc:1514] Processing DeleteTablet for tablet f0193f84567242b381b600e71006fccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:42:19 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:50874
I20260924 10:42:19.951468 16334 tablet_replica.cc:330] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4: stopping tablet replica
I20260924 10:42:19.951550 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 15897
I20260924 10:42:19.951556 16335 tablet_replica.cc:330] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: stopping tablet replica
I20260924 10:42:19.951829 16334 raft_consensus.cc:2229] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:42:19.952286 16334 raft_consensus.cc:2258] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:42:19.953614 16334 ts_tablet_manager.cc:1905] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:42:19.956391 16334 ts_tablet_manager.cc:1918] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:42:19.956497 16334 log.cc:1196] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/f0193f84567242b381b600e71006fccc
I20260924 10:42:19.956887 16334 ts_tablet_manager.cc:1939] T f0193f84567242b381b600e71006fccc P 7c09872922854c8a82c4c5d6f11683b4: Deleting consensus metadata
I20260924 10:42:19.958137 16333 tablet_replica.cc:330] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: stopping tablet replica
I20260924 10:42:19.958153 15822 catalog_manager.cc:4841] TS 7c09872922854c8a82c4c5d6f11683b4 (127.2.8.3:46731): tablet f0193f84567242b381b600e71006fccc (table test-table [id=5c04ddde7011474a932ebb70259ccab4]) successfully deleted
I20260924 10:42:19.958434 16333 raft_consensus.cc:2229] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:42:19.958729 16333 raft_consensus.cc:2258] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 10:42:19.959105 15822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.8.1:33271 (error 108)
I20260924 10:42:19.959710 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 16031
I20260924 10:42:19.961452 16307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.961157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:33271 (local address 127.0.0.1:56212)
0924 10:42:19.961293 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:19.961366 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 10:42:19.961638 15822 catalog_manager.cc:4584] TS 4748cb5a7f29474b8f48929756786b22 (127.2.8.1:33271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0193f84567242b381b600e71006fccc: Network error: Client connection negotiation failed: client connection to 127.2.8.1:33271: connect: Connection refused (error 111)
I20260924 10:42:19.965129 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 16165
I20260924 10:42:19.965401 16307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:19.965071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:41243 (local address 127.0.0.1:49910)
0924 10:42:19.965233 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:19.965301 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.2:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:42:19.970106 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 15803
I20260924 10:42:19.976333 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:46121
--webserver_interface=127.2.8.62
--webserver_port=37245
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:46121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:20.084476 16337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:20.084726 16337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:20.084805 16337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:20.088716 16337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:42:20.088773 16337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:20.088792 16337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:20.088809 16337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:42:20.088871 16337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:42:20.094058 16337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:46121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:46121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=37245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:20.095074 16337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:20.096079 16337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:20.108800 16343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:20.108803 16346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:20.108803 16344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:20.108857 16337 server_base.cc:1034] running on GCE node
I20260924 10:42:20.109702 16337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:20.110854 16337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:20.112020 16337 hybrid_clock.cc:648] HybridClock initialized: now 1790246540111998 us; error 32 us; skew 500 ppm
Sep 24 10:42:20 dist-test-slave-j38h krb5kdc[15790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246540, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:20.114804 16337 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:42:20.115657 16337 webserver.cc:460] Webserver started at http://127.2.8.62:37245/ using document root <none> and password file <none>
I20260924 10:42:20.116219 16337 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:20.116326 16337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:20.121217 16337 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:42:20.123318 16353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:20.124519 16337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:42:20.124678 16337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "c05fa2ac1c2745419924fd67f50e0c49"
format_stamp: "Formatted at 2026-09-24 10:42:18 on dist-test-slave-j38h"
server_key: "991de6c99f1bc4ef3ca1160aad9348f1"
server_key_iv: "3da720042aedc76285cbc9f7ee8d02d6"
server_key_version: "encryptionkey@0"
I20260924 10:42:20.125316 16337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:20.145709 16337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:20.149400 16337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:20.149618 16337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:20.156832 16337 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:46121
I20260924 10:42:20.157158 16405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:46121 every 8 connection(s)
I20260924 10:42:20.158022 16337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:42:20.162294 16406 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:42:20.162294 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 16337
I20260924 10:42:20.162727 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:33271
--local_ip_for_outbound_sockets=127.2.8.1
--tserver_master_addrs=127.2.8.62:46121
--webserver_port=35313
--webserver_interface=127.2.8.1
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:42:20.164060 16406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49: Bootstrap starting.
I20260924 10:42:20.187325 16406 log.cc:824] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:20.196596 16406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:42:20.197108 16406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49: Bootstrap complete.
I20260924 10:42:20.201249 16406 raft_consensus.cc:348] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } }
I20260924 10:42:20.201668 16406 raft_consensus.cc:729] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05fa2ac1c2745419924fd67f50e0c49, State: Initialized, Role: FOLLOWER
I20260924 10:42:20.202157 16406 consensus_queue.cc:260] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } }
I20260924 10:42:20.202273 16406 raft_consensus.cc:388] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:42:20.202378 16406 raft_consensus.cc:482] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:42:20.202502 16406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:42:20.204427 16406 raft_consensus.cc:504] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } }
I20260924 10:42:20.204775 16406 leader_election.cc:302] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c05fa2ac1c2745419924fd67f50e0c49; no voters:
I20260924 10:42:20.205152 16406 leader_election.cc:288] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:42:20.205291 16410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:42:20.205540 16410 raft_consensus.cc:686] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 2 LEADER]: Becoming Leader. State: Replica: c05fa2ac1c2745419924fd67f50e0c49, State: Running, Role: LEADER
I20260924 10:42:20.205914 16410 consensus_queue.cc:237] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } }
I20260924 10:42:20.206393 16406 sys_catalog.cc:564] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:42:20.207384 16411 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c05fa2ac1c2745419924fd67f50e0c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } } }
I20260924 10:42:20.207499 16411 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:20.207418 16412 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c05fa2ac1c2745419924fd67f50e0c49. Latest consensus state: current_term: 2 leader_uuid: "c05fa2ac1c2745419924fd67f50e0c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } } }
I20260924 10:42:20.207561 16412 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:20.208767 16419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:42:20.212597 16419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:42:20.213222 16419 catalog_manager.cc:1259] Loaded cluster ID: fadab5abd7174a50aad7e64f795cd7be
I20260924 10:42:20.213304 16419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:42:20.214462 16419 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 10:42:20.217284 16419 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 10:42:20.221833 16419 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:42:20.222440 16419 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49: Loaded TSK: 0
I20260924 10:42:20.223486 16419 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:42:20.291412 16408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:20.291694 16408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:20.291738 16408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:20.295744 16408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:20.295799 16408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:20.295886 16408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:42:20.301626 16408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:33271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=35313
--tserver_master_addrs=127.2.8.62:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:20.302618 16408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:20.303619 16408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:20.313391 16434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:20.313375 16436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:20.313375 16433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:20.313970 16408 server_base.cc:1034] running on GCE node
I20260924 10:42:20.314392 16408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:20.315062 16408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:20.316251 16408 hybrid_clock.cc:648] HybridClock initialized: now 1790246540316225 us; error 43 us; skew 500 ppm
Sep 24 10:42:20 dist-test-slave-j38h krb5kdc[15790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246540, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:20.319217 16408 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:42:20.320112 16408 webserver.cc:460] Webserver started at http://127.2.8.1:35313/ using document root <none> and password file <none>
I20260924 10:42:20.320696 16408 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:20.320806 16408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:20.324793 16408 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:42:20.327018 16443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:20.328428 16408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:42:20.328569 16408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "4748cb5a7f29474b8f48929756786b22"
format_stamp: "Formatted at 2026-09-24 10:42:19 on dist-test-slave-j38h"
server_key: "c880b5494e43068b180e29f8c09b0970"
server_key_iv: "fdb52bf81d8577135a57b62519c39edc"
server_key_version: "encryptionkey@0"
I20260924 10:42:20.329205 16408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:20.339958 16408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:20.343245 16408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:20.343441 16408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:20.343999 16408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:20.345160 16450 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 10:42:20.349313 16408 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:42:20.349393 16408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260924 10:42:20.349466 16408 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:42:20.351087 16408 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:42:20.351141 16408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 10:42:20.351387 16450 tablet_bootstrap.cc:492] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: Bootstrap starting.
I20260924 10:42:20.361797 16408 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:33271
I20260924 10:42:20.361968 16557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:33271 every 8 connection(s)
I20260924 10:42:20.362967 16408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:42:20.369900 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 16408
I20260924 10:42:20.370417 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:41243
--local_ip_for_outbound_sockets=127.2.8.2
--tserver_master_addrs=127.2.8.62:46121
--webserver_port=46211
--webserver_interface=127.2.8.2
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:42:20.371697 16450 log.cc:824] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:42:20 dist-test-slave-j38h krb5kdc[15790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246540, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:20.380003 16450 tablet_bootstrap.cc:492] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:42:20.380431 16450 tablet_bootstrap.cc:492] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: Bootstrap complete.
I20260924 10:42:20.380488 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.365112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:35041)
0924 10:42:20.365607 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:20.365647 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:20.366420 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:20.367487 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:20.367495 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:20.367900 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:20.368340 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:20.368353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.369714 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.369717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:20.370758 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:20.370768 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.371429 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.371464 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:20.371655 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.372269 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:20.372294 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:20.374181 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:20.377021 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:20.377027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:20.377040 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:20.377391 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:20.377833 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:20.377836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:20.377838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:20.377994 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:20.378762 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:20.378768 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:20.379038 (+ 270us) client_negotiation.cc:769] Sending connection context
0924 10:42:20.379229 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 10:42:20.379533 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":135,"threads_started":1}
I20260924 10:42:20.380895 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.365099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:35041 (local address 127.2.8.62:46121)
0924 10:42:20.365438 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:20.365457 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:20.365757 (+ 300us) server_negotiation.cc:407] Connection header received
0924 10:42:20.366833 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:20.366843 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:20.367118 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:20.367435 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:20.368503 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.369526 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.370877 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.371321 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.371496 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.374392 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:20.374402 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:20.374421 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:20.374458 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:20.376857 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:20.377523 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:20.377531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:20.377540 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:20.377716 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:20.378113 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:20.378119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:20.378124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:20.378629 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:20.378776 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:20.379407 (+ 631us) server_negotiation.cc:299] Negotiation successful
0924 10:42:20.379755 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":119,"threads_started":1}
I20260924 10:42:20.381798 16450 ts_tablet_manager.cc:1397] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.007s
I20260924 10:42:20.383458 16558 heartbeater.cc:347] Connected to a master server at 127.2.8.62:46121
I20260924 10:42:20.383689 16558 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:20.384213 16558 heartbeater.cc:511] Master 127.2.8.62:46121 requested a full tablet report, sending...
I20260924 10:42:20.384404 16450 raft_consensus.cc:348] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } } peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } }
I20260924 10:42:20.384928 16450 raft_consensus.cc:729] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4748cb5a7f29474b8f48929756786b22, State: Initialized, Role: FOLLOWER
I20260924 10:42:20.385452 16450 consensus_queue.cc:260] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } } peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } }
I20260924 10:42:20.386381 16450 ts_tablet_manager.cc:1428] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:42:20.388453 16370 ts_manager.cc:194] Registered new tserver with Master: 4748cb5a7f29474b8f48929756786b22 (127.2.8.1:33271)
I20260924 10:42:20.392468 16370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:35041
I20260924 10:42:20.395315 16558 heartbeater.cc:503] Master 127.2.8.62:46121 was elected leader, sending a full tablet report...
Sep 24 10:42:20 dist-test-slave-j38h krb5kdc[15790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246540, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for kudu/127.2.8.1@KRBTEST.COM
I20260924 10:42:20.402065 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.391776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:33271 (local address 127.0.0.1:33266)
0924 10:42:20.392304 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:20.392350 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:20.392484 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:20.393050 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:20.393056 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:20.393516 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:20.393621 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:20.393628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.394622 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.394627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:20.395248 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:20.395257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.396149 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.396166 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:20.396274 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.396923 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:20.396939 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:20.398336 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:20.400362 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:20.400370 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:20.400374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:20.400592 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:20.400926 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:20.400929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:20.400931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:20.401062 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:20.401575 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:20.401581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:20.401685 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 10:42:20.401792 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 10:42:20.401940 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":127,"threads_started":1}
I20260924 10:42:20.402232 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.391822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.2.8.1:33271)
0924 10:42:20.392546 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:20.392561 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:20.392585 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:42:20.392725 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:20.392729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:20.392882 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:20.393046 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:20.393745 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.394507 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.395383 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.395911 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.396027 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.398470 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:20.398480 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:20.398493 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:20.398519 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:20.400252 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:20.400699 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:20.400704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:20.400706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:20.400803 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:20.401157 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:20.401160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:20.401162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:20.401442 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:20.401568 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:20.401915 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 10:42:20.402077 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":628,"thread_start_us":94,"threads_started":1}
I20260924 10:42:20.403363 16492 tablet_service.cc:1514] Processing DeleteTablet for tablet f0193f84567242b381b600e71006fccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:42:19 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:33266
I20260924 10:42:20.404304 16567 tablet_replica.cc:330] stopping tablet replica
I20260924 10:42:20.404595 16567 raft_consensus.cc:2229] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:42:20.404865 16567 raft_consensus.cc:2258] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:42:20.405856 16567 ts_tablet_manager.cc:1905] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:42:20.409605 16567 ts_tablet_manager.cc:1918] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:42:20.409691 16567 log.cc:1196] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal/wals/f0193f84567242b381b600e71006fccc
I20260924 10:42:20.409950 16567 ts_tablet_manager.cc:1939] T f0193f84567242b381b600e71006fccc P 4748cb5a7f29474b8f48929756786b22: Deleting consensus metadata
I20260924 10:42:20.411521 16357 catalog_manager.cc:4841] TS 4748cb5a7f29474b8f48929756786b22 (127.2.8.1:33271): tablet f0193f84567242b381b600e71006fccc (table test-table [id=5c04ddde7011474a932ebb70259ccab4]) successfully deleted
W20260924 10:42:20.488641 16562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:20.488886 16562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:20.488927 16562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:20.492841 16562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:20.492901 16562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:20.492957 16562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:42:20.497969 16562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:41243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=46211
--tserver_master_addrs=127.2.8.62:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:20.498883 16562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:20.499797 16562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:20.509222 16572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:20.509338 16562 server_base.cc:1034] running on GCE node
W20260924 10:42:20.509233 16573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:20.509233 16575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:20.510316 16562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:20.510883 16562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:20.512066 16562 hybrid_clock.cc:648] HybridClock initialized: now 1790246540512054 us; error 35 us; skew 500 ppm
Sep 24 10:42:20 dist-test-slave-j38h krb5kdc[15790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246540, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:20.514714 16562 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:42:20.515590 16562 webserver.cc:460] Webserver started at http://127.2.8.2:46211/ using document root <none> and password file <none>
I20260924 10:42:20.516139 16562 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:20.516247 16562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:20.519910 16562 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:42:20.522058 16582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:20.523273 16562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:42:20.523399 16562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "999bc3d14dc64566b08e1dd16f85f7c0"
format_stamp: "Formatted at 2026-09-24 10:42:19 on dist-test-slave-j38h"
server_key: "0366a5916366bd9d90622c27766b4a26"
server_key_iv: "1b04e1c8a6e45863632131e40e294daf"
server_key_version: "encryptionkey@0"
I20260924 10:42:20.523921 16562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:20.549532 16562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:20.552408 16562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:20.552621 16562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:20.553233 16562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:20.554311 16589 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 10:42:20.558710 16562 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:42:20.558784 16562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260924 10:42:20.558866 16562 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:42:20.560490 16562 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:42:20.560559 16562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 10:42:20.560842 16589 tablet_bootstrap.cc:492] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: Bootstrap starting.
I20260924 10:42:20.566987 16589 log.cc:824] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:20.570817 16562 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:41243
I20260924 10:42:20.570894 16697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:41243 every 8 connection(s)
I20260924 10:42:20.571669 16562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:42:20.575531 16589 tablet_bootstrap.cc:492] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:42:20.575946 16589 tablet_bootstrap.cc:492] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: Bootstrap complete.
I20260924 10:42:20.576285 16589 ts_tablet_manager.cc:1397] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260924 10:42:20.578210 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 16562
I20260924 10:42:20.578596 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:46731
--local_ip_for_outbound_sockets=127.2.8.3
--tserver_master_addrs=127.2.8.62:46121
--webserver_port=35005
--webserver_interface=127.2.8.3
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:42:20.579111 16589 raft_consensus.cc:348] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } } peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } }
I20260924 10:42:20.579520 16589 raft_consensus.cc:729] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999bc3d14dc64566b08e1dd16f85f7c0, State: Initialized, Role: FOLLOWER
I20260924 10:42:20.580034 16589 consensus_queue.cc:260] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } } peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } }
I20260924 10:42:20.581238 16589 ts_tablet_manager.cc:1428] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
Sep 24 10:42:20 dist-test-slave-j38h krb5kdc[15790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246540, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:20.586645 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.573486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:60047 (local address 127.2.8.62:46121)
0924 10:42:20.573637 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:20.573642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:20.574228 (+ 586us) server_negotiation.cc:407] Connection header received
0924 10:42:20.575305 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:20.575307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:20.575355 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:20.575427 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:20.576802 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.577418 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.579034 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.579458 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.579576 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.582699 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:20.582706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:20.582709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:20.582743 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:20.584498 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:20.584969 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:20.585000 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:20.585011 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:20.585111 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:20.585431 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:20.585436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:20.585439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:20.585701 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:20.585812 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:20.586258 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 10:42:20.586416 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:42:20.587319 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.573585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:60047)
0924 10:42:20.574050 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:20.574101 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:20.575085 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:20.575583 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:20.575591 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:20.576044 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:20.576659 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:20.576677 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.577566 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.577570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:20.578855 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:20.578865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.579701 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.579721 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:20.579966 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.581070 (+ 1104us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:20.581086 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:20.582522 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:20.584615 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:20.584620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:20.584630 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:20.584868 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:20.585207 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:20.585209 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:20.585210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:20.585333 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:20.585811 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:20.585815 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:20.586018 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 10:42:20.586172 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 10:42:20.586401 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":121,"threads_started":1}
I20260924 10:42:20.588466 16698 heartbeater.cc:347] Connected to a master server at 127.2.8.62:46121
I20260924 10:42:20.588724 16698 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:20.589363 16698 heartbeater.cc:511] Master 127.2.8.62:46121 requested a full tablet report, sending...
I20260924 10:42:20.591320 16370 ts_manager.cc:194] Registered new tserver with Master: 999bc3d14dc64566b08e1dd16f85f7c0 (127.2.8.2:41243)
I20260924 10:42:20.592530 16370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:60047
I20260924 10:42:20.594729 16698 heartbeater.cc:503] Master 127.2.8.62:46121 was elected leader, sending a full tablet report...
Sep 24 10:42:20 dist-test-slave-j38h krb5kdc[15790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246540, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for kudu/127.2.8.2@KRBTEST.COM
I20260924 10:42:20.600983 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.592185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:41243 (local address 127.0.0.1:48682)
0924 10:42:20.592343 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:20.592357 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:20.592421 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:20.593009 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:20.593012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:20.593212 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:20.593292 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:20.593297 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.594256 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.594262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:20.594780 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:20.594809 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.595507 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.595520 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:20.595618 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.596179 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:20.596196 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:20.597459 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:20.599384 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:20.599387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:20.599390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:20.599615 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:20.599915 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:20.599919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:20.599921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:20.600001 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:20.600476 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:20.600479 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:20.600575 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 10:42:20.600687 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 10:42:20.600820 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 10:42:20.601110 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.592299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.2.8.2:41243)
0924 10:42:20.592563 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:20.592575 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:20.592598 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:42:20.592661 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:20.592666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:20.592862 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:20.593025 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:20.593402 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.594138 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.594908 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.595404 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.595513 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.597581 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:20.597589 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:20.597612 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:20.597640 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:20.599242 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:20.599730 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:20.599735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:20.599737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:20.599823 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:20.600106 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:20.600111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:20.600113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:20.600361 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:20.600496 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:20.600766 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 10:42:20.600914 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":110,"threads_started":1}
I20260924 10:42:20.601980 16631 tablet_service.cc:1514] Processing DeleteTablet for tablet f0193f84567242b381b600e71006fccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:42:19 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:48682
I20260924 10:42:20.602985 16704 tablet_replica.cc:330] stopping tablet replica
I20260924 10:42:20.603291 16704 raft_consensus.cc:2229] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:42:20.603596 16704 raft_consensus.cc:2258] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:42:20.604584 16704 ts_tablet_manager.cc:1905] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:42:20.608631 16704 ts_tablet_manager.cc:1918] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:42:20.608722 16704 log.cc:1196] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/f0193f84567242b381b600e71006fccc
I20260924 10:42:20.609093 16704 ts_tablet_manager.cc:1939] T f0193f84567242b381b600e71006fccc P 999bc3d14dc64566b08e1dd16f85f7c0: Deleting consensus metadata
I20260924 10:42:20.610589 16356 catalog_manager.cc:4841] TS 999bc3d14dc64566b08e1dd16f85f7c0 (127.2.8.2:41243): tablet f0193f84567242b381b600e71006fccc (table test-table [id=5c04ddde7011474a932ebb70259ccab4]) successfully deleted
W20260924 10:42:20.695322 16701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:20.695571 16701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:20.695613 16701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:20.699472 16701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:20.699518 16701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:20.699576 16701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:42:20.704689 16701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:46731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=35005
--tserver_master_addrs=127.2.8.62:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:20.705794 16701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:20.706775 16701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:20.716730 16709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:20.716714 16712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:20.716802 16710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:20.717084 16701 server_base.cc:1034] running on GCE node
I20260924 10:42:20.717473 16701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:20.718080 16701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:20.719280 16701 hybrid_clock.cc:648] HybridClock initialized: now 1790246540719269 us; error 35 us; skew 500 ppm
Sep 24 10:42:20 dist-test-slave-j38h krb5kdc[15790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246540, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:20.722196 16701 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:42:20.723042 16701 webserver.cc:460] Webserver started at http://127.2.8.3:35005/ using document root <none> and password file <none>
I20260924 10:42:20.723601 16701 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:20.723714 16701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:20.727810 16701 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:42:20.730118 16719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:20.731341 16701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:42:20.731482 16701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "7c09872922854c8a82c4c5d6f11683b4"
format_stamp: "Formatted at 2026-09-24 10:42:19 on dist-test-slave-j38h"
server_key: "2db522ebd38ed1287cc9b1cacd2c87da"
server_key_iv: "30dd16340e4df5aa480e9ec5b806dcf0"
server_key_version: "encryptionkey@0"
I20260924 10:42:20.732074 16701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:20.750104 16701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:20.753139 16701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:20.753348 16701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:20.753940 16701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:20.754768 16701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:42:20.754838 16701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:20.754908 16701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:20.754956 16701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:20.763943 16701 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:46731
I20260924 10:42:20.764022 16832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:46731 every 8 connection(s)
I20260924 10:42:20.765071 16701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:42:20.765708 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 16701
I20260924 10:42:20.775457 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.766522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.2.8.62:46121)
0924 10:42:20.766698 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:20.766703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:20.766885 (+ 182us) server_negotiation.cc:407] Connection header received
0924 10:42:20.766987 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:20.766992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:20.767048 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:20.767163 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:20.768041 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.768904 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.770152 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.770477 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.770577 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.772193 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:20.772197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:20.772200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:20.772227 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:20.773756 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:20.774188 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:20.774191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:20.774193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:20.774276 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:20.774562 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:20.774564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:20.774565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:20.774736 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:20.774955 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:20.775147 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 10:42:20.775278 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 10:42:20 dist-test-slave-j38h krb5kdc[15790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246540, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:20.780627 16836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.767345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:58675 (local address 127.2.8.62:46121)
0924 10:42:20.767590 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:20.767596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:20.768459 (+ 863us) server_negotiation.cc:407] Connection header received
0924 10:42:20.770085 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:20.770088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:20.770166 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:20.770264 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:20.771847 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.772600 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.773982 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.774267 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.774374 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.777405 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:20.777411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:20.777413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:20.777439 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:20.778823 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:20.779314 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:20.779317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:20.779318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:20.779384 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:20.779686 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:20.779688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:20.779689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:20.779853 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:20.779923 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:20.780406 (+ 483us) server_negotiation.cc:299] Negotiation successful
0924 10:42:20.780500 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260924 10:42:20.781598 16837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.767817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:58675)
0924 10:42:20.768326 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:20.768367 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:20.769847 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:20.770428 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:20.770435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:20.770957 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:20.771655 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:20.771677 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.772724 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.772729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:20.773851 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:20.773862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.774369 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.774426 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:20.774672 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.775322 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:20.775345 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:20.777225 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:20.778934 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:20.778939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:20.778953 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:20.779220 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:20.779457 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:20.779459 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:20.779460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:20.779605 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:20.779934 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:20.779938 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:20.780187 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 10:42:20.780348 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 10:42:20.780620 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":181,"threads_started":1}
I20260924 10:42:20.782692 16833 heartbeater.cc:347] Connected to a master server at 127.2.8.62:46121
I20260924 10:42:20.782943 16833 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:20.783428 16833 heartbeater.cc:511] Master 127.2.8.62:46121 requested a full tablet report, sending...
I20260924 10:42:20.784444 16370 ts_manager.cc:194] Registered new tserver with Master: 7c09872922854c8a82c4c5d6f11683b4 (127.2.8.3:46731)
I20260924 10:42:20.785121 16370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:58675
I20260924 10:42:20.789469 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:42:20.794183 16836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.790152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.2.8.62:46121)
0924 10:42:20.790284 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:20.790288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:20.790513 (+ 225us) server_negotiation.cc:407] Connection header received
0924 10:42:20.790657 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:20.790660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:20.790752 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:20.790837 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:42:20.791123 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.791628 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.792525 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.792752 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.792830 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.793176 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:42:20.793818 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:42:20.793930 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:20.794009 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 10:42:20.794066 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:42:20.794893 16370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:42:20.798522 16370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:42:20.806226 16631 tablet_service.cc:1467] Processing CreateTablet for tablet 153cc2ac25724a7a9c10596a310bea06 (DEFAULT_TABLE table=test-table [id=69d674e65766462d8c14a2a0aeb315e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:42:20.806269 16492 tablet_service.cc:1467] Processing CreateTablet for tablet 153cc2ac25724a7a9c10596a310bea06 (DEFAULT_TABLE table=test-table [id=69d674e65766462d8c14a2a0aeb315e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:42:20.806876 16631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153cc2ac25724a7a9c10596a310bea06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:20.806876 16492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153cc2ac25724a7a9c10596a310bea06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:20.809752 16450 tablet_bootstrap.cc:492] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22: Bootstrap starting.
I20260924 10:42:20.810271 16589 tablet_bootstrap.cc:492] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0: Bootstrap starting.
I20260924 10:42:20.810518 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.805486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:46731 (local address 127.0.0.1:41726)
0924 10:42:20.805649 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:20.805664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:20.805738 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:20.806747 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:20.806750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:20.806767 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:20.806874 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:20.806881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.808164 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.808170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:20.809120 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:20.809129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.810081 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.810085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:20.810197 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.810246 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 10:42:20.810316 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 10:42:20.810396 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 10:42:20.810649 16450 tablet_bootstrap.cc:654] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:20.810894 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.805843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.2.8.3:46731)
0924 10:42:20.806223 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:20.806239 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:20.806267 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:42:20.806349 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:20.806354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:20.806588 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:20.806776 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:20.807005 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.808039 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.809248 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.809967 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.810078 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.810563 (+ 485us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:20.810625 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 10:42:20.810728 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":205,"threads_started":1}
I20260924 10:42:20.811221 16589 tablet_bootstrap.cc:654] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:20.812139 16450 tablet_bootstrap.cc:492] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22: No bootstrap required, opened a new log
I20260924 10:42:20.812219 16450 ts_tablet_manager.cc:1397] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:42:20.812611 16450 raft_consensus.cc:348] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } } peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } }
I20260924 10:42:20.812692 16450 raft_consensus.cc:374] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:20.812702 16589 tablet_bootstrap.cc:492] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0: No bootstrap required, opened a new log
I20260924 10:42:20.812719 16450 raft_consensus.cc:729] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4748cb5a7f29474b8f48929756786b22, State: Initialized, Role: FOLLOWER
I20260924 10:42:20.812772 16589 ts_tablet_manager.cc:1397] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:42:20.812853 16450 consensus_queue.cc:260] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } } peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } }
I20260924 10:42:20.813134 16450 ts_tablet_manager.cc:1428] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:42:20.813175 16589 raft_consensus.cc:348] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } } peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } }
I20260924 10:42:20.813271 16589 raft_consensus.cc:374] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:20.813303 16589 raft_consensus.cc:729] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999bc3d14dc64566b08e1dd16f85f7c0, State: Initialized, Role: FOLLOWER
I20260924 10:42:20.813410 16589 consensus_queue.cc:260] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } } peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } }
I20260924 10:42:20.813648 16589 ts_tablet_manager.cc:1428] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:42:20.814633 16767 tablet_service.cc:1467] Processing CreateTablet for tablet 153cc2ac25724a7a9c10596a310bea06 (DEFAULT_TABLE table=test-table [id=69d674e65766462d8c14a2a0aeb315e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:42:20.815601 16767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153cc2ac25724a7a9c10596a310bea06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:20.822395 16845 tablet_bootstrap.cc:492] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4: Bootstrap starting.
W20260924 10:42:20.822553 16699 tablet.cc:2369] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:42:20.824352 16845 tablet_bootstrap.cc:654] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:20.825444 16845 log.cc:824] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:20.827296 16845 tablet_bootstrap.cc:492] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4: No bootstrap required, opened a new log
I20260924 10:42:20.827521 16845 ts_tablet_manager.cc:1397] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:42:20.830575 16845 raft_consensus.cc:348] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } } peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } }
I20260924 10:42:20.830821 16845 raft_consensus.cc:374] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:20.830878 16845 raft_consensus.cc:729] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c09872922854c8a82c4c5d6f11683b4, State: Initialized, Role: FOLLOWER
I20260924 10:42:20.831314 16845 consensus_queue.cc:260] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } } peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } }
I20260924 10:42:20.831988 16833 heartbeater.cc:503] Master 127.2.8.62:46121 was elected leader, sending a full tablet report...
I20260924 10:42:20.832391 16845 ts_tablet_manager.cc:1428] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:42:20.846488 16564 raft_consensus.cc:482] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:42:20.846796 16564 raft_consensus.cc:504] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } } peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } }
I20260924 10:42:20.848330 16564 leader_election.cc:288] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c09872922854c8a82c4c5d6f11683b4 (127.2.8.3:46731), 999bc3d14dc64566b08e1dd16f85f7c0 (127.2.8.2:41243)
I20260924 10:42:20.853606 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.848507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:46731 (local address 127.2.8.1:43845)
0924 10:42:20.848657 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:20.848676 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:20.848781 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:20.849185 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:20.849191 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:20.849242 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:20.849424 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:20.849432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.850599 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.850603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:20.851805 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:20.851819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.853010 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.853014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:20.853166 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.853230 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 10:42:20.853314 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 10:42:20.853414 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":29}
I20260924 10:42:20.854029 16842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.848658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:43845 (local address 127.2.8.3:46731)
0924 10:42:20.848798 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:20.848803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:20.848821 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:42:20.849013 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:20.849018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:20.849079 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:20.849166 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:20.849575 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.850489 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.851962 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.852712 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.852830 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.852918 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:20.853791 (+ 873us) server_negotiation.cc:299] Negotiation successful
0924 10:42:20.853885 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:42:20.854545 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.848587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:41243 (local address 127.2.8.1:51139)
0924 10:42:20.848965 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:20.849005 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:20.849162 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:20.849575 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:20.849580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:20.849625 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:20.849742 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:20.849748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.850788 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.850791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:20.852331 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:20.852365 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.853670 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.853674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:20.854091 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.854164 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 10:42:20.854302 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 10:42:20.854380 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":39,"thread_start_us":107,"threads_started":1}
I20260924 10:42:20.854564 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.848811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:51139 (local address 127.2.8.2:41243)
0924 10:42:20.848942 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:20.848947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:20.849077 (+ 130us) server_negotiation.cc:407] Connection header received
0924 10:42:20.849292 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:20.849295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:20.849362 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:20.849429 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:20.849900 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.850670 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.852513 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.853529 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.853653 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.853839 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:20.854326 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 10:42:20.854414 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:42:20.855406 16651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153cc2ac25724a7a9c10596a310bea06" candidate_uuid: "4748cb5a7f29474b8f48929756786b22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" is_pre_election: true
I20260924 10:42:20.855865 16651 raft_consensus.cc:2454] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4748cb5a7f29474b8f48929756786b22 in term 0.
I20260924 10:42:20.856092 16787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153cc2ac25724a7a9c10596a310bea06" candidate_uuid: "4748cb5a7f29474b8f48929756786b22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c09872922854c8a82c4c5d6f11683b4" is_pre_election: true
I20260924 10:42:20.856372 16787 raft_consensus.cc:2454] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4748cb5a7f29474b8f48929756786b22 in term 0.
I20260924 10:42:20.856421 16446 leader_election.cc:302] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4748cb5a7f29474b8f48929756786b22, 999bc3d14dc64566b08e1dd16f85f7c0; no voters:
I20260924 10:42:20.856789 16564 raft_consensus.cc:2781] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:42:20.856899 16564 raft_consensus.cc:482] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:42:20.856954 16564 raft_consensus.cc:3037] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:20.858405 16564 raft_consensus.cc:504] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } } peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } }
I20260924 10:42:20.858922 16564 leader_election.cc:288] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 1 election: Requested vote from peers 7c09872922854c8a82c4c5d6f11683b4 (127.2.8.3:46731), 999bc3d14dc64566b08e1dd16f85f7c0 (127.2.8.2:41243)
I20260924 10:42:20.859308 16787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153cc2ac25724a7a9c10596a310bea06" candidate_uuid: "4748cb5a7f29474b8f48929756786b22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c09872922854c8a82c4c5d6f11683b4"
I20260924 10:42:20.859318 16651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153cc2ac25724a7a9c10596a310bea06" candidate_uuid: "4748cb5a7f29474b8f48929756786b22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "999bc3d14dc64566b08e1dd16f85f7c0"
I20260924 10:42:20.859443 16787 raft_consensus.cc:3037] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:20.859442 16651 raft_consensus.cc:3037] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:20.860514 16787 raft_consensus.cc:2454] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4748cb5a7f29474b8f48929756786b22 in term 1.
I20260924 10:42:20.860549 16651 raft_consensus.cc:2454] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4748cb5a7f29474b8f48929756786b22 in term 1.
I20260924 10:42:20.860843 16447 leader_election.cc:302] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4748cb5a7f29474b8f48929756786b22, 7c09872922854c8a82c4c5d6f11683b4; no voters:
I20260924 10:42:20.861058 16564 raft_consensus.cc:2781] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:42:20.861274 16564 raft_consensus.cc:686] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 LEADER]: Becoming Leader. State: Replica: 4748cb5a7f29474b8f48929756786b22, State: Running, Role: LEADER
I20260924 10:42:20.861515 16564 consensus_queue.cc:237] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } } peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } }
W20260924 10:42:20.864298 16559 tablet.cc:2369] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:42:20.864215 16370 catalog_manager.cc:5495] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4748cb5a7f29474b8f48929756786b22 (127.2.8.1). New cstate: current_term: 1 leader_uuid: "4748cb5a7f29474b8f48929756786b22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } health_report { overall_health: HEALTHY } } }
I20260924 10:42:20.882288 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.878716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.2.8.1:33271)
0924 10:42:20.878852 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:20.878855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:20.878893 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:42:20.879103 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:20.879106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:20.879165 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:20.879261 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:42:20.879509 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.880064 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.880884 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:20.881213 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:20.881299 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:20.881531 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:42:20.881968 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:42:20.882074 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:20.882124 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 10:42:20.882174 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 10:42:20.886253 16787 raft_consensus.cc:1261] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4 [term 1 FOLLOWER]: Refusing update from remote peer 4748cb5a7f29474b8f48929756786b22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:42:20.886250 16651 raft_consensus.cc:1261] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0 [term 1 FOLLOWER]: Refusing update from remote peer 4748cb5a7f29474b8f48929756786b22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:42:20.887017 16849 consensus_queue.cc:1035] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:42:20.887214 16564 consensus_queue.cc:1035] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:42:20.893549 16853 mvcc.cc:205] Tried to move back new op lower bound from 7332849831463768064 to 7332849831371317248. Current Snapshot: MvccSnapshot[applied={T|T < 7332849831463768064}]
I20260924 10:42:20.893678 16852 mvcc.cc:205] Tried to move back new op lower bound from 7332849831463768064 to 7332849831371317248. Current Snapshot: MvccSnapshot[applied={T|T < 7332849831463768064}]
I20260924 10:42:20.895408 16854 mvcc.cc:205] Tried to move back new op lower bound from 7332849831463768064 to 7332849831371317248. Current Snapshot: MvccSnapshot[applied={T|T < 7332849831463768064}]
I20260924 10:42:20.901310 16368 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:42:20.901455 16368 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:42:20.903810 16368 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49: Sending DeleteTablet for 3 replicas of tablet 153cc2ac25724a7a9c10596a310bea06
I20260924 10:42:20.904443 16767 tablet_service.cc:1514] Processing DeleteTablet for tablet 153cc2ac25724a7a9c10596a310bea06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:42:20 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:41726
I20260924 10:42:20.904604 16631 tablet_service.cc:1514] Processing DeleteTablet for tablet 153cc2ac25724a7a9c10596a310bea06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:42:20 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:48682
I20260924 10:42:20.904609 16492 tablet_service.cc:1514] Processing DeleteTablet for tablet 153cc2ac25724a7a9c10596a310bea06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:42:20 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:33266
I20260924 10:42:20.904774 16704 tablet_replica.cc:330] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0: stopping tablet replica
I20260924 10:42:20.904791 16567 tablet_replica.cc:330] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22: stopping tablet replica
I20260924 10:42:20.904896 16567 raft_consensus.cc:2229] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:42:20.904901 16704 raft_consensus.cc:2229] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:42:20.905052 16861 tablet_replica.cc:330] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4: stopping tablet replica
I20260924 10:42:20.905053 16704 raft_consensus.cc:2258] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:42:20.905189 16567 raft_consensus.cc:2258] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:42:20.905335 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 16408
I20260924 10:42:20.905336 16861 raft_consensus.cc:2229] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:42:20.905625 16861 raft_consensus.cc:2258] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:42:20.906368 16704 ts_tablet_manager.cc:1905] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:42:20.908425 16861 ts_tablet_manager.cc:1905] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:42:20.908939 16704 ts_tablet_manager.cc:1918] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:42:20.909047 16704 log.cc:1196] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/153cc2ac25724a7a9c10596a310bea06
I20260924 10:42:20.909299 16704 ts_tablet_manager.cc:1939] T 153cc2ac25724a7a9c10596a310bea06 P 999bc3d14dc64566b08e1dd16f85f7c0: Deleting consensus metadata
I20260924 10:42:20.910166 16356 catalog_manager.cc:4841] TS 999bc3d14dc64566b08e1dd16f85f7c0 (127.2.8.2:41243): tablet 153cc2ac25724a7a9c10596a310bea06 (table test-table [id=69d674e65766462d8c14a2a0aeb315e6]) successfully deleted
I20260924 10:42:20.911098 16861 ts_tablet_manager.cc:1918] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:42:20.911183 16861 log.cc:1196] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/153cc2ac25724a7a9c10596a310bea06
I20260924 10:42:20.911494 16861 ts_tablet_manager.cc:1939] T 153cc2ac25724a7a9c10596a310bea06 P 7c09872922854c8a82c4c5d6f11683b4: Deleting consensus metadata
W20260924 10:42:20.912560 16357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.8.1:33271 (error 108)
I20260924 10:42:20.913121 16357 catalog_manager.cc:4841] TS 7c09872922854c8a82c4c5d6f11683b4 (127.2.8.3:46731): tablet 153cc2ac25724a7a9c10596a310bea06 (table test-table [id=69d674e65766462d8c14a2a0aeb315e6]) successfully deleted
I20260924 10:42:20.913226 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 16562
I20260924 10:42:20.913987 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:20.913437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:33271 (local address 127.0.0.1:33284)
0924 10:42:20.913814 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:20.913890 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:33271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285}
W20260924 10:42:20.914304 16357 catalog_manager.cc:4584] TS 4748cb5a7f29474b8f48929756786b22 (127.2.8.1:33271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 153cc2ac25724a7a9c10596a310bea06: Network error: Client connection negotiation failed: client connection to 127.2.8.1:33271: connect: Connection refused (error 111)
I20260924 10:42:20.918871 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 16701
I20260924 10:42:20.925239 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 16337
I20260924 10:42:20.931578 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:46121
--webserver_interface=127.2.8.62
--webserver_port=37245
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:46121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:21.040679 16863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:21.040952 16863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:21.041074 16863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:21.045228 16863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:42:21.045289 16863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:21.045310 16863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:21.045326 16863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:42:21.045390 16863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:42:21.050730 16863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:46121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:46121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=37245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:21.051774 16863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:21.052800 16863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:21.061350 16868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:21.061401 16863 server_base.cc:1034] running on GCE node
W20260924 10:42:21.061383 16871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:21.061379 16869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:21.062175 16863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:21.063232 16863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:21.064428 16863 hybrid_clock.cc:648] HybridClock initialized: now 1790246541064417 us; error 30 us; skew 500 ppm
Sep 24 10:42:21 dist-test-slave-j38h krb5kdc[15790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246541, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:21.067226 16863 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:42:21.068094 16863 webserver.cc:460] Webserver started at http://127.2.8.62:37245/ using document root <none> and password file <none>
I20260924 10:42:21.068584 16863 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:21.068640 16863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:21.072320 16863 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260924 10:42:21.074333 16878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:21.075497 16863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:42:21.075595 16863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "c05fa2ac1c2745419924fd67f50e0c49"
format_stamp: "Formatted at 2026-09-24 10:42:18 on dist-test-slave-j38h"
server_key: "991de6c99f1bc4ef3ca1160aad9348f1"
server_key_iv: "3da720042aedc76285cbc9f7ee8d02d6"
server_key_version: "encryptionkey@0"
I20260924 10:42:21.076089 16863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:21.095430 16863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:21.098670 16863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:21.098817 16863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:21.105501 16863 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:46121
I20260924 10:42:21.105839 16930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:46121 every 8 connection(s)
I20260924 10:42:21.106565 16863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:42:21.107823 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 16863
I20260924 10:42:21.108316 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:33271
--local_ip_for_outbound_sockets=127.2.8.1
--tserver_master_addrs=127.2.8.62:46121
--webserver_port=35313
--webserver_interface=127.2.8.1
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:42:21.111582 16931 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:42:21.113263 16931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49: Bootstrap starting.
I20260924 10:42:21.131150 16931 log.cc:824] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:21.142320 16931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:42:21.142712 16931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49: Bootstrap complete.
I20260924 10:42:21.147028 16931 raft_consensus.cc:348] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } }
I20260924 10:42:21.147408 16931 raft_consensus.cc:729] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05fa2ac1c2745419924fd67f50e0c49, State: Initialized, Role: FOLLOWER
I20260924 10:42:21.147946 16931 consensus_queue.cc:260] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } }
I20260924 10:42:21.148066 16931 raft_consensus.cc:388] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:42:21.148156 16931 raft_consensus.cc:482] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:42:21.148274 16931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 2 FOLLOWER]: Advancing to term 3
I20260924 10:42:21.150170 16931 raft_consensus.cc:504] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } }
I20260924 10:42:21.150493 16931 leader_election.cc:302] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c05fa2ac1c2745419924fd67f50e0c49; no voters:
I20260924 10:42:21.150818 16931 leader_election.cc:288] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 10:42:21.150923 16935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 10:42:21.151130 16935 raft_consensus.cc:686] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [term 3 LEADER]: Becoming Leader. State: Replica: c05fa2ac1c2745419924fd67f50e0c49, State: Running, Role: LEADER
I20260924 10:42:21.151410 16935 consensus_queue.cc:237] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } }
I20260924 10:42:21.152029 16931 sys_catalog.cc:564] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:42:21.152333 16935 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c05fa2ac1c2745419924fd67f50e0c49. Latest consensus state: current_term: 3 leader_uuid: "c05fa2ac1c2745419924fd67f50e0c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } } }
I20260924 10:42:21.152448 16935 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:21.152818 16939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:42:21.153005 16936 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c05fa2ac1c2745419924fd67f50e0c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05fa2ac1c2745419924fd67f50e0c49" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 46121 } } }
I20260924 10:42:21.153231 16936 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05fa2ac1c2745419924fd67f50e0c49 [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:21.158259 16939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:42:21.159132 16939 catalog_manager.cc:1259] Loaded cluster ID: fadab5abd7174a50aad7e64f795cd7be
I20260924 10:42:21.159195 16939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 10:42:21.160159 16939 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3f800fe9ad google::LogMessage::Fail() at ??:0
@ 0x7f3f8010188a google::LogMessage::SendToLog() at ??:0
@ 0x7f3f800fe4de google::LogMessage::Flush() at ??:0
@ 0x7f3f800ffa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3f8aa3dfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3f8aa3ed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3f8aa3b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f3f8aa7f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3f823503d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3f82350c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3f823523f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3f82341512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3f839856db start_thread at ??:0
@ 0x7f3f7ebfc61f clone at ??:0
W20260924 10:42:21.249024 16933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:21.249305 16933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:21.249351 16933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:21.253400 16933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:21.253460 16933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:21.253546 16933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:42:21.259016 16933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:33271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=35313
--tserver_master_addrs=127.2.8.62:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:21.260128 16933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:21.261250 16933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:21.270951 16972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:21.271068 16975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:21.270951 16973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:21.271411 16933 server_base.cc:1034] running on GCE node
I20260924 10:42:21.271848 16933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:21.272531 16933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:21.273710 16933 hybrid_clock.cc:648] HybridClock initialized: now 1790246541273682 us; error 47 us; skew 500 ppm
Sep 24 10:42:21 dist-test-slave-j38h krb5kdc[15790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246541, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:21.276499 16933 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:42:21.277446 16933 webserver.cc:460] Webserver started at http://127.2.8.1:35313/ using document root <none> and password file <none>
I20260924 10:42:21.278004 16933 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:21.278064 16933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:21.282423 16933 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:42:21.284797 16982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:21.286216 16933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:42:21.286350 16933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "4748cb5a7f29474b8f48929756786b22"
format_stamp: "Formatted at 2026-09-24 10:42:19 on dist-test-slave-j38h"
server_key: "c880b5494e43068b180e29f8c09b0970"
server_key_iv: "fdb52bf81d8577135a57b62519c39edc"
server_key_version: "encryptionkey@0"
I20260924 10:42:21.286892 16933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:21.300477 16933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:21.304003 16933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:21.304224 16933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:21.304841 16933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:21.306134 16989 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 10:42:21.310252 16933 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:42:21.310322 16933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260924 10:42:21.310392 16933 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:42:21.312057 16933 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:42:21.312114 16933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 10:42:21.312582 16989 tablet_bootstrap.cc:492] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22: Bootstrap starting.
I20260924 10:42:21.329543 16933 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:33271
I20260924 10:42:21.331007 16933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:42:21.332456 17096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:33271 every 8 connection(s)
I20260924 10:42:21.336551 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 16933
I20260924 10:42:21.337185 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:41243
--local_ip_for_outbound_sockets=127.2.8.2
--tserver_master_addrs=127.2.8.62:46121
--webserver_port=46211
--webserver_interface=127.2.8.2
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:42:21.337491 16989 log.cc:824] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:21.347102 16989 tablet_bootstrap.cc:492] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:42:21.347558 16989 tablet_bootstrap.cc:492] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22: Bootstrap complete.
I20260924 10:42:21.365226 16989 ts_tablet_manager.cc:1397] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22: Time spent bootstrapping tablet: real 0.053s user 0.027s sys 0.007s
I20260924 10:42:21.370007 16989 raft_consensus.cc:348] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } } peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } }
I20260924 10:42:21.370814 16989 raft_consensus.cc:729] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4748cb5a7f29474b8f48929756786b22, State: Initialized, Role: FOLLOWER
I20260924 10:42:21.371618 16989 consensus_queue.cc:260] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } } peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } }
I20260924 10:42:21.372690 16989 ts_tablet_manager.cc:1428] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260924 10:42:21.424815 17099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:42:21.333713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:41937)
0924 10:42:21.336453 (+ 2740us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:21.336515 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:21.337605 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:21.423883 (+ 86278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2504,"thread_start_us":162,"threads_started":1}
W20260924 10:42:21.425370 17097 heartbeater.cc:644] Failed to heartbeat to 127.2.8.62:46121 (0 consecutive failures): Network error: Failed to ping master at 127.2.8.62:46121: Client connection negotiation failed: client connection to 127.2.8.62:46121: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:42:21.426218 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:21.425955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:47655)
0924 10:42:21.426100 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:21.426163 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:42:21.426939 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:21.426743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:54655)
0924 10:42:21.426847 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:21.426891 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 10:42:21.427166 17097 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:42:21.482388 17101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:21.482718 17101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:21.482797 17101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:21.486756 17101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:21.486846 17101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:21.486935 17101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:42:21.492283 17101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:41243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=46211
--tserver_master_addrs=127.2.8.62:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:21.493346 17101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:21.494336 17101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:21.503739 17110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:21.503739 17107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:21.503789 17108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:21.503887 17101 server_base.cc:1034] running on GCE node
I20260924 10:42:21.504382 17101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:21.505018 17101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:21.506219 17101 hybrid_clock.cc:648] HybridClock initialized: now 1790246541506189 us; error 50 us; skew 500 ppm
Sep 24 10:42:21 dist-test-slave-j38h krb5kdc[15790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246541, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:21.508926 17101 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:42:21.509797 17101 webserver.cc:460] Webserver started at http://127.2.8.2:46211/ using document root <none> and password file <none>
I20260924 10:42:21.510354 17101 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:21.510445 17101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:21.514272 17101 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 10:42:21.516376 17117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:21.517611 17101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:42:21.517745 17101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "999bc3d14dc64566b08e1dd16f85f7c0"
format_stamp: "Formatted at 2026-09-24 10:42:19 on dist-test-slave-j38h"
server_key: "0366a5916366bd9d90622c27766b4a26"
server_key_iv: "1b04e1c8a6e45863632131e40e294daf"
server_key_version: "encryptionkey@0"
I20260924 10:42:21.518271 17101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:21.560304 17101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:21.563724 17101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:21.563947 17101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:21.564513 17101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:21.565348 17101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:42:21.565394 17101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:21.565434 17101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:21.565449 17101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:21.574273 17101 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:41243
I20260924 10:42:21.574295 17230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:41243 every 8 connection(s)
I20260924 10:42:21.575316 17101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:42:21.579226 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:21.577529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:60081)
0924 10:42:21.577973 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:21.578081 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":114,"threads_started":1}
W20260924 10:42:21.579799 17231 heartbeater.cc:644] Failed to heartbeat to 127.2.8.62:46121 (0 consecutive failures): Network error: Failed to ping master at 127.2.8.62:46121: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
I20260924 10:42:21.580415 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:21.580179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:54887)
0924 10:42:21.580303 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:21.580355 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 10:42:21.581159 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:21.580914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:35507)
0924 10:42:21.581054 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:21.581107 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:42:21.581375 17231 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:42:21.584376 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 17101
I20260924 10:42:21.584789 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:46731
--local_ip_for_outbound_sockets=127.2.8.3
--tserver_master_addrs=127.2.8.62:46121
--webserver_port=35005
--webserver_interface=127.2.8.3
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:21.696663 17234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:21.696921 17234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:21.696964 17234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:21.701057 17234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:21.701107 17234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:21.701165 17234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:42:21.706228 17234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:46731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=35005
--tserver_master_addrs=127.2.8.62:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:21.707157 17234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:21.708091 17234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:21.717450 17239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:21.717410 17242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:21.717531 17240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:21.717572 17234 server_base.cc:1034] running on GCE node
I20260924 10:42:21.718039 17234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:21.718638 17234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:21.719803 17234 hybrid_clock.cc:648] HybridClock initialized: now 1790246541719770 us; error 44 us; skew 500 ppm
Sep 24 10:42:21 dist-test-slave-j38h krb5kdc[15790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246541, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:21.722816 17234 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:42:21.723657 17234 webserver.cc:460] Webserver started at http://127.2.8.3:35005/ using document root <none> and password file <none>
I20260924 10:42:21.724200 17234 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:21.724260 17234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:21.728031 17234 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:42:21.730211 17249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:21.731473 17234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:42:21.731653 17234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "7c09872922854c8a82c4c5d6f11683b4"
format_stamp: "Formatted at 2026-09-24 10:42:19 on dist-test-slave-j38h"
server_key: "2db522ebd38ed1287cc9b1cacd2c87da"
server_key_iv: "30dd16340e4df5aa480e9ec5b806dcf0"
server_key_version: "encryptionkey@0"
I20260924 10:42:21.732191 17234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:21.752461 17234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:21.755398 17234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:21.755616 17234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:21.756213 17234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:21.757085 17234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:42:21.757158 17234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:21.757236 17234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:21.757280 17234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:21.766078 17234 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:46731
I20260924 10:42:21.766099 17362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:46731 every 8 connection(s)
I20260924 10:42:21.767062 17234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:42:21.771015 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:21.769294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:40563)
0924 10:42:21.769778 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:21.769881 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":111,"threads_started":1}
I20260924 10:42:21.771257 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 17234
W20260924 10:42:21.771554 17363 heartbeater.cc:644] Failed to heartbeat to 127.2.8.62:46121 (0 consecutive failures): Network error: Failed to ping master at 127.2.8.62:46121: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
I20260924 10:42:21.772205 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:21.771989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:42979)
0924 10:42:21.772103 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:21.772157 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 10:42:21.772823 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:21.772608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:42843)
0924 10:42:21.772731 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:21.772776 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 10:42:21.773061 17363 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:42:22.428407 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:22.427888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:47201)
0924 10:42:22.428206 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:22.428298 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":84,"threads_started":1}
I20260924 10:42:22.582492 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:22.582001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:57917)
0924 10:42:22.582301 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:22.582365 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":107,"threads_started":1}
I20260924 10:42:22.774071 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:22.773662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:57625)
0924 10:42:22.773915 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:22.773973 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260924 10:42:22.796105 17370 raft_consensus.cc:482] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:42:22.796344 17370 raft_consensus.cc:504] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } } peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } }
I20260924 10:42:22.797621 17370 leader_election.cc:288] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c09872922854c8a82c4c5d6f11683b4 (127.2.8.3:46731), 999bc3d14dc64566b08e1dd16f85f7c0 (127.2.8.2:41243)
Sep 24 10:42:22 dist-test-slave-j38h krb5kdc[15790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246541, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.3@KRBTEST.COM
Sep 24 10:42:22 dist-test-slave-j38h krb5kdc[15790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246541, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.2@KRBTEST.COM
I20260924 10:42:22.811463 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:22.797789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:46731 (local address 127.2.8.1:46443)
0924 10:42:22.798068 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:22.798084 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:22.798258 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:22.800441 (+ 2183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:22.800444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:22.801070 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:22.801528 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:22.801543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:22.802808 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:22.802818 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:22.803752 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:22.803762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:22.804462 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:22.804479 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:22.804765 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:22.805484 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:22.805496 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:22.806944 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:22.809219 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:22.809226 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:22.809239 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:22.809482 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:22.809833 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:22.809837 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:22.809839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:22.809966 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:22.810575 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:22.810582 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:22.810798 (+ 216us) client_negotiation.cc:769] Sending connection context
0924 10:42:22.810978 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 10:42:22.811278 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":141,"spinlock_wait_cycles":24192}
I20260924 10:42:22.811506 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:22.798590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:46443 (local address 127.2.8.3:46731)
0924 10:42:22.799062 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:22.799084 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:22.799124 (+ 40us) server_negotiation.cc:407] Connection header received
0924 10:42:22.799781 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:22.799788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:22.800144 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:22.800480 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:22.801711 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:22.802665 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:22.803928 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:22.804350 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:22.804532 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:22.807163 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:22.807168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:22.807180 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:22.807207 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:22.809100 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:22.809600 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:22.809607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:22.809614 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:22.809733 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:22.810072 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:22.810075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:22.810077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:22.810455 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:22.810601 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:22.811096 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 10:42:22.811344 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":125,"threads_started":1}
I20260924 10:42:22.811885 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:22.797900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:41243 (local address 127.2.8.1:40777)
0924 10:42:22.798274 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:22.798288 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:22.798401 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:22.800378 (+ 1977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:22.800387 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:22.801077 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:22.801527 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:22.801543 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:22.802816 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:22.802819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:22.803750 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:22.803759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:22.804460 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:22.804479 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:22.804754 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:22.805440 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:22.805456 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:22.807311 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:22.809575 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:22.809578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:22.809580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:22.809779 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:22.810110 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:22.810114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:22.810116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:22.810201 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:22.810748 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:22.810752 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:22.810841 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 10:42:22.810978 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 10:42:22.811672 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":144,"thread_start_us":89,"threads_started":1}
I20260924 10:42:22.812108 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:22.798085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:40777 (local address 127.2.8.2:41243)
0924 10:42:22.798697 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:22.798724 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:22.798773 (+ 49us) server_negotiation.cc:407] Connection header received
0924 10:42:22.799570 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:22.799577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:22.799937 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:22.800304 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:22.801721 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:22.802662 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:22.803873 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:22.804351 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:22.804538 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:22.807521 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:22.807529 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:22.807544 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:22.807580 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:22.809437 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:22.809888 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:22.809893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:22.809900 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:22.810004 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:22.810296 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:22.810299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:22.810301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:22.810644 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:22.810777 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:22.811093 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 10:42:22.811368 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":93,"threads_started":1}
W20260924 10:42:22.814244 16985 leader_election.cc:341] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 999bc3d14dc64566b08e1dd16f85f7c0 (127.2.8.2:41243): Not found: Tablet not found: 153cc2ac25724a7a9c10596a310bea06
W20260924 10:42:22.814448 16986 leader_election.cc:341] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c09872922854c8a82c4c5d6f11683b4 (127.2.8.3:46731): Not found: Tablet not found: 153cc2ac25724a7a9c10596a310bea06
I20260924 10:42:22.814561 16986 leader_election.cc:302] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4748cb5a7f29474b8f48929756786b22; no voters: 7c09872922854c8a82c4c5d6f11683b4, 999bc3d14dc64566b08e1dd16f85f7c0
I20260924 10:42:22.814821 17370 raft_consensus.cc:2726] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:42:23.429596 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:23.429209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:52155)
0924 10:42:23.429449 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:23.429503 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260924 10:42:23.583580 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:23.583144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:45239)
0924 10:42:23.583421 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:23.583489 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":68,"threads_started":1}
I20260924 10:42:23.775276 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:23.774863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:48389)
0924 10:42:23.775116 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:23.775185 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260924 10:42:24.431062 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:24.430583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:58725)
0924 10:42:24.430908 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:24.430967 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":114,"threads_started":1}
I20260924 10:42:24.571193 17378 raft_consensus.cc:482] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:42:24.571317 17378 raft_consensus.cc:504] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } } peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } }
I20260924 10:42:24.571990 17378 leader_election.cc:288] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c09872922854c8a82c4c5d6f11683b4 (127.2.8.3:46731), 999bc3d14dc64566b08e1dd16f85f7c0 (127.2.8.2:41243)
W20260924 10:42:24.572767 16986 leader_election.cc:341] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c09872922854c8a82c4c5d6f11683b4 (127.2.8.3:46731): Not found: Tablet not found: 153cc2ac25724a7a9c10596a310bea06
W20260924 10:42:24.573035 16985 leader_election.cc:341] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 999bc3d14dc64566b08e1dd16f85f7c0 (127.2.8.2:41243): Not found: Tablet not found: 153cc2ac25724a7a9c10596a310bea06
I20260924 10:42:24.573114 16985 leader_election.cc:302] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4748cb5a7f29474b8f48929756786b22; no voters: 7c09872922854c8a82c4c5d6f11683b4, 999bc3d14dc64566b08e1dd16f85f7c0
I20260924 10:42:24.573251 17378 raft_consensus.cc:2726] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:42:24.584704 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:24.584253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:56375)
0924 10:42:24.584509 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:24.584602 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":110,"threads_started":1}
I20260924 10:42:24.776446 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:24.776010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:55079)
0924 10:42:24.776290 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:24.776346 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":98,"threads_started":1}
I20260924 10:42:25.432327 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:25.431853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:54249)
0924 10:42:25.432170 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:25.432230 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":84,"threads_started":1}
I20260924 10:42:25.585812 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:25.585415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:38559)
0924 10:42:25.585660 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:25.585717 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":69,"threads_started":1}
I20260924 10:42:25.777783 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:25.777329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:58919)
0924 10:42:25.777612 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:25.777681 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
I20260924 10:42:26.433745 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:26.433251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:60369)
0924 10:42:26.433541 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:26.433599 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":84,"threads_started":1}
I20260924 10:42:26.587090 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:26.586603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:34445)
0924 10:42:26.586903 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:26.586956 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":108,"threads_started":1}
I20260924 10:42:26.779234 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:26.778786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:33567)
0924 10:42:26.779065 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:26.779138 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
I20260924 10:42:27.435056 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:27.434614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:53591)
0924 10:42:27.434902 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:27.434961 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
I20260924 10:42:27.588387 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:27.587941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:39755)
0924 10:42:27.588193 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:27.588282 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
I20260924 10:42:27.780435 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:27.780015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:51273)
0924 10:42:27.780287 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:27.780344 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
I20260924 10:42:27.906179 17390 raft_consensus.cc:482] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:42:27.906313 17390 raft_consensus.cc:504] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } } peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } }
I20260924 10:42:27.906803 17390 leader_election.cc:288] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c09872922854c8a82c4c5d6f11683b4 (127.2.8.3:46731), 999bc3d14dc64566b08e1dd16f85f7c0 (127.2.8.2:41243)
W20260924 10:42:27.907572 16985 leader_election.cc:341] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 999bc3d14dc64566b08e1dd16f85f7c0 (127.2.8.2:41243): Not found: Tablet not found: 153cc2ac25724a7a9c10596a310bea06
W20260924 10:42:27.907671 16986 leader_election.cc:341] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c09872922854c8a82c4c5d6f11683b4 (127.2.8.3:46731): Not found: Tablet not found: 153cc2ac25724a7a9c10596a310bea06
I20260924 10:42:27.907792 16986 leader_election.cc:302] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4748cb5a7f29474b8f48929756786b22; no voters: 7c09872922854c8a82c4c5d6f11683b4, 999bc3d14dc64566b08e1dd16f85f7c0
I20260924 10:42:27.907974 17390 raft_consensus.cc:2726] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:42:28.436242 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:28.435843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:58669)
0924 10:42:28.436075 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:28.436147 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260924 10:42:28.589599 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:28.589147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:56735)
0924 10:42:28.589447 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:28.589506 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":80,"threads_started":1}
I20260924 10:42:28.781783 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:28.781283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:45011)
0924 10:42:28.781633 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:28.781688 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
I20260924 10:42:29.437316 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:29.436925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:48275)
0924 10:42:29.437170 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:29.437223 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
I20260924 10:42:29.590835 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:29.590407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:40145)
0924 10:42:29.590693 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:29.590746 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":108,"threads_started":1}
I20260924 10:42:29.782989 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:29.782561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:60289)
0924 10:42:29.782829 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:29.782890 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
I20260924 10:42:30.438588 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:30.438179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:33957)
0924 10:42:30.438435 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:30.438493 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":79,"threads_started":1}
I20260924 10:42:30.592129 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:30.591670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:53087)
0924 10:42:30.591959 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:30.592021 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":89,"threads_started":1}
I20260924 10:42:30.784178 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:30.783778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:56539)
0924 10:42:30.784028 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:30.784085 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
I20260924 10:42:30.891855 17400 raft_consensus.cc:482] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:42:30.891983 17400 raft_consensus.cc:504] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c09872922854c8a82c4c5d6f11683b4" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46731 } } peers { permanent_uuid: "999bc3d14dc64566b08e1dd16f85f7c0" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 41243 } } peers { permanent_uuid: "4748cb5a7f29474b8f48929756786b22" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33271 } }
I20260924 10:42:30.892357 17400 leader_election.cc:288] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c09872922854c8a82c4c5d6f11683b4 (127.2.8.3:46731), 999bc3d14dc64566b08e1dd16f85f7c0 (127.2.8.2:41243)
W20260924 10:42:30.893219 16986 leader_election.cc:341] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c09872922854c8a82c4c5d6f11683b4 (127.2.8.3:46731): Not found: Tablet not found: 153cc2ac25724a7a9c10596a310bea06
W20260924 10:42:30.893328 16985 leader_election.cc:341] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 999bc3d14dc64566b08e1dd16f85f7c0 (127.2.8.2:41243): Not found: Tablet not found: 153cc2ac25724a7a9c10596a310bea06
I20260924 10:42:30.893388 16985 leader_election.cc:302] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4748cb5a7f29474b8f48929756786b22; no voters: 7c09872922854c8a82c4c5d6f11683b4, 999bc3d14dc64566b08e1dd16f85f7c0
I20260924 10:42:30.893532 17400 raft_consensus.cc:2726] T 153cc2ac25724a7a9c10596a310bea06 P 4748cb5a7f29474b8f48929756786b22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 10:42:31.439802 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:31.439399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:46399)
0924 10:42:31.439650 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:31.439707 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260924 10:42:31.593482 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:31.593067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:42391)
0924 10:42:31.593316 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:31.593383 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260924 10:42:31.785467 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:31.784955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:52045)
0924 10:42:31.785296 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:31.785372 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":88,"threads_started":1}
I20260924 10:42:32.440923 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:32.440521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:58045)
0924 10:42:32.440774 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:32.440826 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":92,"threads_started":1}
I20260924 10:42:32.594689 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:32.594287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:54545)
0924 10:42:32.594546 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:32.594600 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260924 10:42:32.786773 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:32.786282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:38119)
0924 10:42:32.786626 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:32.786680 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":90,"threads_started":1}
I20260924 10:42:33.442103 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:33.441703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:41453)
0924 10:42:33.441941 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:33.441998 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260924 10:42:33.595759 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:33.595363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:60207)
0924 10:42:33.595614 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:33.595667 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
I20260924 10:42:33.788019 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:33.787535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:55779)
0924 10:42:33.787831 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:33.787905 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":91,"threads_started":1}
I20260924 10:42:34.443281 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:34.442870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:35093)
0924 10:42:34.443124 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:34.443179 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":88,"threads_started":1}
I20260924 10:42:34.596941 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:34.596536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:40349)
0924 10:42:34.596789 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:34.596849 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260924 10:42:34.789301 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:34.788838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:39741)
0924 10:42:34.789137 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:34.789195 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":85,"threads_started":1}
I20260924 10:42:35.444564 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:35.444113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:34633)
0924 10:42:35.444357 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:35.444424 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260924 10:42:35.598136 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:35.597714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:52449)
0924 10:42:35.597980 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:35.598044 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
I20260924 10:42:35.790643 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:35.790189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.3:60233)
0924 10:42:35.790460 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:35.790522 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":89,"threads_started":1}
I20260924 10:42:36.445916 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:36.445510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.1:58739)
0924 10:42:36.445770 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:36.445823 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":87,"threads_started":1}
I20260924 10:42:36.599452 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:36.599063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:46121 (local address 127.2.8.2:54891)
0924 10:42:36.599306 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:36.599358 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
I20260924 10:42:36.778988 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 16933
I20260924 10:42:36.784788 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 17101
I20260924 10:42:36.790426 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 17234
2026-09-24T10:42:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19191 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:42:36 dist-test-slave-j38h krb5kdc[17420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:42:36 dist-test-slave-j38h krb5kdc[17420](info): set up 2 sockets
Sep 24 10:42:36 dist-test-slave-j38h krb5kdc[17420](info): commencing operation
krb5kdc: starting...
W20260924 10:42:37.841302 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:42:37 dist-test-slave-j38h krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:42:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:42:37.963562 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:45591
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:45591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:38.077107 17433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:38.077389 17433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:38.077440 17433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:38.081440 17433 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 10:42:38.081490 17433 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 10:42:38.081509 17433 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 10:42:38.081526 17433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:42:38.081547 17433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:38.081568 17433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:38.081593 17433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:42:38.081611 17433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:42:38.086669 17433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:45591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:45591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:38.087594 17433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:38.088514 17433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:38.096807 17441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:38.096822 17439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:38.096992 17433 server_base.cc:1034] running on GCE node
W20260924 10:42:38.096829 17438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:38.097684 17433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:38.098690 17433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:38.099866 17433 hybrid_clock.cc:648] HybridClock initialized: now 1790246558099848 us; error 29 us; skew 500 ppm
Sep 24 10:42:38 dist-test-slave-j38h krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246558, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:38.102661 17433 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:42:38.103533 17433 webserver.cc:460] Webserver started at http://127.2.8.62:46579/ using document root <none> and password file <none>
I20260924 10:42:38.104066 17433 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:38.104125 17433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:38.104362 17433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:42:38.106357 17433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "47797fad5d7e4511b1040bcbaa4b7048"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "5bac52b6c910de4bc9ca86c2bd38229d"
server_key_iv: "a0e8cf6d8d60a5ada6de1b2fb3c658e5"
server_key_version: "encryptionkey@0"
I20260924 10:42:38.106911 17433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "47797fad5d7e4511b1040bcbaa4b7048"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "5bac52b6c910de4bc9ca86c2bd38229d"
server_key_iv: "a0e8cf6d8d60a5ada6de1b2fb3c658e5"
server_key_version: "encryptionkey@0"
I20260924 10:42:38.110585 17433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 10:42:38.113150 17448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:38.114435 17433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:42:38.114560 17433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "47797fad5d7e4511b1040bcbaa4b7048"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "5bac52b6c910de4bc9ca86c2bd38229d"
server_key_iv: "a0e8cf6d8d60a5ada6de1b2fb3c658e5"
server_key_version: "encryptionkey@0"
I20260924 10:42:38.114665 17433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:38.134635 17433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:38.137799 17433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:38.138044 17433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:38.145538 17433 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:45591
I20260924 10:42:38.145843 17500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:45591 every 8 connection(s)
I20260924 10:42:38.146595 17433 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:42:38.149490 17501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:38.149837 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 17433
I20260924 10:42:38.149999 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:42:38.150314 2080 external_mini_cluster.cc:1428] Setting key 7186789ce33af461e3e0ace8971208b7
I20260924 10:42:38.155623 17501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Bootstrap starting.
I20260924 10:42:38.158229 17501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:38.159168 17501 log.cc:824] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:42:38 dist-test-slave-j38h krb5kdc[17420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:38.161370 17501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: No bootstrap required, opened a new log
I20260924 10:42:38.164783 17501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } }
I20260924 10:42:38.165040 17501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:38.165115 17501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47797fad5d7e4511b1040bcbaa4b7048, State: Initialized, Role: FOLLOWER
I20260924 10:42:38.165388 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:38.151677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57726 (local address 127.2.8.62:45591)
0924 10:42:38.152105 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:38.152120 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:38.152156 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:42:38.152815 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:38.152831 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:38.153225 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:38.153520 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:38.154476 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.155451 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.156615 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.157154 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.157287 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:38.160306 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:38.160311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:38.160324 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:38.160351 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:38.162258 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:38.162746 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:38.162752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:38.162758 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:38.162883 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:38.163200 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:38.163205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:38.163207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:38.163616 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:38.163835 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:38.164167 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 10:42:38.164458 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":147,"threads_started":1}
I20260924 10:42:38.165650 17501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } }
I20260924 10:42:38.165832 17501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:42:38.165921 17501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:42:38.166021 17501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:38.167107 17501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } }
I20260924 10:42:38.167460 17501 leader_election.cc:302] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47797fad5d7e4511b1040bcbaa4b7048; no voters:
I20260924 10:42:38.167788 17501 leader_election.cc:288] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:42:38.167922 17506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:42:38.168185 17506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 1 LEADER]: Becoming Leader. State: Replica: 47797fad5d7e4511b1040bcbaa4b7048, State: Running, Role: LEADER
I20260924 10:42:38.168548 17506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } }
I20260924 10:42:38.169121 17501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:42:38.170315 17507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47797fad5d7e4511b1040bcbaa4b7048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } } }
I20260924 10:42:38.170306 17508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47797fad5d7e4511b1040bcbaa4b7048. Latest consensus state: current_term: 1 leader_uuid: "47797fad5d7e4511b1040bcbaa4b7048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } } }
I20260924 10:42:38.170449 17508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:38.170456 17507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:38.170917 17515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:42:38.173930 17515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:42:38.179258 17515 catalog_manager.cc:1347] Generated new cluster ID: fc2ea8eb88ae49978761aca9042f1b9e
I20260924 10:42:38.179339 17515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:42:38.188059 17515 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:42:38.188895 17515 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:42:38.193122 17515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Generated new TSK 0
I20260924 10:42:38.193871 17515 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:42:38.232031 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:45591
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:38.341310 17527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:38.341583 17527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:38.341625 17527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:38.345571 17527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:38.345626 17527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:38.345685 17527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:42:38.350871 17527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:38.351819 17527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:38.352797 17527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:38.362291 17533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:38.362270 17532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:38.362483 17527 server_base.cc:1034] running on GCE node
W20260924 10:42:38.362270 17535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:38.362977 17527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:38.363632 17527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:38.364821 17527 hybrid_clock.cc:648] HybridClock initialized: now 1790246558364799 us; error 45 us; skew 500 ppm
Sep 24 10:42:38 dist-test-slave-j38h krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246558, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:38.367677 17527 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:42:38.368515 17527 webserver.cc:460] Webserver started at http://127.2.8.1:44557/ using document root <none> and password file <none>
I20260924 10:42:38.369143 17527 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:38.369237 17527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:38.369565 17527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:42:38.371893 17527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "e28a7ed37a6f41738f3d48228a072c2d"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "5741d069547885c52b100e3e17e97476"
server_key_iv: "cb9d77d7189dcceff5e9131d9963baf0"
server_key_version: "encryptionkey@0"
I20260924 10:42:38.372534 17527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "e28a7ed37a6f41738f3d48228a072c2d"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "5741d069547885c52b100e3e17e97476"
server_key_iv: "cb9d77d7189dcceff5e9131d9963baf0"
server_key_version: "encryptionkey@0"
I20260924 10:42:38.376329 17527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:42:38.378937 17542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:38.380277 17527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 10:42:38.380425 17527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "e28a7ed37a6f41738f3d48228a072c2d"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "5741d069547885c52b100e3e17e97476"
server_key_iv: "cb9d77d7189dcceff5e9131d9963baf0"
server_key_version: "encryptionkey@0"
I20260924 10:42:38.380541 17527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:38.392815 17527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:38.395730 17527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:38.395921 17527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:38.396468 17527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:38.397323 17527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:42:38.397392 17527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:38.397461 17527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:38.397511 17527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:38.406033 17527 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:34223
I20260924 10:42:38.406065 17655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:34223 every 8 connection(s)
I20260924 10:42:38.406857 17527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:42:38.407923 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 17527
I20260924 10:42:38.408087 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:42:38.408380 2080 external_mini_cluster.cc:1428] Setting key 7d6bfa437e52afef013a24143dc35e5c
Sep 24 10:42:38 dist-test-slave-j38h krb5kdc[17420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246558, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
I20260924 10:42:38.420429 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:38.408861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:38747 (local address 127.2.8.62:45591)
0924 10:42:38.409159 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:38.409165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:38.409883 (+ 718us) server_negotiation.cc:407] Connection header received
0924 10:42:38.410785 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:38.410791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:38.410853 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:38.410947 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:38.412122 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.412651 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.413875 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.414144 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.414258 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:38.416677 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:38.416680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:38.416683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:38.416715 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:38.418324 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:38.418827 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:38.418832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:38.418834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:38.418934 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:38.419272 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:38.419276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:38.419278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:38.419455 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:38.419568 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:38.420102 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 10:42:38.420247 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260924 10:42:38.421252 17658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:38.408962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:38747)
0924 10:42:38.409738 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:38.409788 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:38.410590 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:38.411110 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:38.411116 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:38.411466 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:38.411972 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:38.411985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.412778 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.412781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:38.413748 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:38.413757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.414244 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.414295 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:38.414469 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:38.415076 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:38.415091 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:38.416498 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:38.418424 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:38.418428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:38.418440 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:38.418697 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:38.419035 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:38.419038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:38.419041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:38.419173 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:38.419568 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:38.419575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:38.419835 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 10:42:38.420041 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 10:42:38.420359 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":132,"threads_started":1}
I20260924 10:42:38.422487 17656 heartbeater.cc:347] Connected to a master server at 127.2.8.62:45591
I20260924 10:42:38.422734 17656 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:38.423287 17656 heartbeater.cc:511] Master 127.2.8.62:45591 requested a full tablet report, sending...
I20260924 10:42:38.425032 17465 ts_manager.cc:194] Registered new tserver with Master: e28a7ed37a6f41738f3d48228a072c2d (127.2.8.1:34223)
I20260924 10:42:38.426699 17465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:38747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:42:38.436275 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:45591
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:38.544862 17661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:38.545143 17661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:38.545187 17661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:38.549136 17661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:38.549187 17661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:38.549245 17661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:42:38.554455 17661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:38.555472 17661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:38.556474 17661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:38.565858 17669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:38.565846 17667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:38.565868 17666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:38.566177 17661 server_base.cc:1034] running on GCE node
I20260924 10:42:38.566576 17661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:38.567240 17661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:38.568397 17661 hybrid_clock.cc:648] HybridClock initialized: now 1790246558568370 us; error 42 us; skew 500 ppm
Sep 24 10:42:38 dist-test-slave-j38h krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246558, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:38.571491 17661 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:42:38.572391 17661 webserver.cc:460] Webserver started at http://127.2.8.2:39115/ using document root <none> and password file <none>
I20260924 10:42:38.572938 17661 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:38.573051 17661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:38.573297 17661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:42:38.575266 17661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "1a23cf3ba12b40eb949ba189063e3fd7"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "3ad746a293de0c9bb70a4206bd1604f8"
server_key_iv: "ccac7ddd65b64fcc2337a3e05d95e91d"
server_key_version: "encryptionkey@0"
I20260924 10:42:38.575817 17661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "1a23cf3ba12b40eb949ba189063e3fd7"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "3ad746a293de0c9bb70a4206bd1604f8"
server_key_iv: "ccac7ddd65b64fcc2337a3e05d95e91d"
server_key_version: "encryptionkey@0"
I20260924 10:42:38.579455 17661 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 10:42:38.581903 17676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:38.583039 17661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:42:38.583180 17661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "1a23cf3ba12b40eb949ba189063e3fd7"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "3ad746a293de0c9bb70a4206bd1604f8"
server_key_iv: "ccac7ddd65b64fcc2337a3e05d95e91d"
server_key_version: "encryptionkey@0"
I20260924 10:42:38.583287 17661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:38.599647 17661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:38.603101 17661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:38.603319 17661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:38.603915 17661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:38.604746 17661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:42:38.604831 17661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:38.604909 17661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:38.604949 17661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:38.614737 17661 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:44075
I20260924 10:42:38.614756 17789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:44075 every 8 connection(s)
I20260924 10:42:38.615654 17661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:42:38.622431 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 17661
I20260924 10:42:38.622557 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:42:38.622833 2080 external_mini_cluster.cc:1428] Setting key 10fd6c88b9f426b19d20682c973c2ed2
Sep 24 10:42:38 dist-test-slave-j38h krb5kdc[17420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246558, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:38.629891 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:38.617695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:33301 (local address 127.2.8.62:45591)
0924 10:42:38.617842 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:38.617846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:38.618897 (+ 1051us) server_negotiation.cc:407] Connection header received
0924 10:42:38.619850 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:38.619853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:38.619908 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:38.620029 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:38.621206 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.621735 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.623225 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.623519 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.623643 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:38.626103 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:38.626108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:38.626113 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:38.626145 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:38.627647 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:38.628272 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:38.628275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:38.628276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:38.628347 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:38.628736 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:38.628739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:38.628740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:38.628911 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:38.629040 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:38.629670 (+ 630us) server_negotiation.cc:299] Negotiation successful
0924 10:42:38.629772 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:42:38.631166 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:38.618046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:33301)
0924 10:42:38.618749 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:38.618788 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:38.619652 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:38.620209 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:38.620216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:38.620580 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:38.621061 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:38.621075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.621858 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.621861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:38.623088 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:38.623099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.623631 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.623684 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:38.623854 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:38.624484 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:38.624500 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:38.625938 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:38.627763 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:38.627772 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:38.627790 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:38.628170 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:38.628434 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:38.628440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:38.628445 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:38.628637 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:38.629043 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:38.629051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:38.629389 (+ 338us) client_negotiation.cc:769] Sending connection context
0924 10:42:38.629625 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 10:42:38.630006 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":150,"threads_started":1}
I20260924 10:42:38.632458 17790 heartbeater.cc:347] Connected to a master server at 127.2.8.62:45591
I20260924 10:42:38.632725 17790 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
I20260924 10:42:38.633343 17790 heartbeater.cc:511] Master 127.2.8.62:45591 requested a full tablet report, sending...
I20260924 10:42:38.634347 17465 ts_manager.cc:194] Registered new tserver with Master: 1a23cf3ba12b40eb949ba189063e3fd7 (127.2.8.2:44075)
I20260924 10:42:38.634922 17465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:33301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:42:38.653548 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:45591
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:38.763303 17795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:38.763599 17795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:38.763684 17795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:38.767746 17795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:38.767835 17795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:38.767930 17795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:42:38.773545 17795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:38.774622 17795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:38.775648 17795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:38.785607 17800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:38.785629 17803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:38.785650 17801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:38.785805 17795 server_base.cc:1034] running on GCE node
I20260924 10:42:38.786405 17795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:38.787091 17795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:38.788268 17795 hybrid_clock.cc:648] HybridClock initialized: now 1790246558788238 us; error 47 us; skew 500 ppm
Sep 24 10:42:38 dist-test-slave-j38h krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246558, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:38.790956 17795 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:42:38.791818 17795 webserver.cc:460] Webserver started at http://127.2.8.3:45119/ using document root <none> and password file <none>
I20260924 10:42:38.792380 17795 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:38.792466 17795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:38.792691 17795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:42:38.794718 17795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "91ba2dee83384729a778675c1c0302f2"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "0dc19bd34d34f25f3b7d2be9c25bd5d9"
server_key_iv: "225f9ea178b2cb44db1c9adca1303438"
server_key_version: "encryptionkey@0"
I20260924 10:42:38.795341 17795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "91ba2dee83384729a778675c1c0302f2"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "0dc19bd34d34f25f3b7d2be9c25bd5d9"
server_key_iv: "225f9ea178b2cb44db1c9adca1303438"
server_key_version: "encryptionkey@0"
I20260924 10:42:38.798920 17795 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:42:38.801355 17810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:38.802613 17795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:42:38.802738 17795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "91ba2dee83384729a778675c1c0302f2"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "0dc19bd34d34f25f3b7d2be9c25bd5d9"
server_key_iv: "225f9ea178b2cb44db1c9adca1303438"
server_key_version: "encryptionkey@0"
I20260924 10:42:38.802845 17795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:38.822964 17795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:38.826026 17795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:38.826239 17795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:38.826814 17795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:38.827613 17795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:42:38.827688 17795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:38.827776 17795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:38.827827 17795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:38.836927 17795 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:36885
I20260924 10:42:38.836951 17923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:36885 every 8 connection(s)
I20260924 10:42:38.837863 17795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:42:38.839833 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 17795
I20260924 10:42:38.839939 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:42:38.840260 2080 external_mini_cluster.cc:1428] Setting key 27ebb1f9671ed875115701c3e871fff3
Sep 24 10:42:38 dist-test-slave-j38h krb5kdc[17420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246558, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:38.852015 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:38.839693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:58455 (local address 127.2.8.62:45591)
0924 10:42:38.839844 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:38.839848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:38.841186 (+ 1338us) server_negotiation.cc:407] Connection header received
0924 10:42:38.842387 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:38.842390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:38.842447 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:38.842504 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:38.844084 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.844607 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.845760 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.845974 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.846067 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:38.848618 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:38.848622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:38.848625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:38.848651 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:38.849977 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:38.850483 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:38.850489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:38.850491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:38.850576 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:38.850921 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:38.850925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:38.850927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:38.851109 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:38.851216 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:38.851726 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 10:42:38.851849 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:42:38.853014 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:38.840164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:58455)
0924 10:42:38.840934 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:38.841055 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:38.842107 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:38.842919 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:38.842928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:38.843428 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:38.843945 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:38.843958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.844726 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.844729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:38.845646 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:38.845655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.846091 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.846134 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:38.846279 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:38.846953 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:38.846969 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:38.848450 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:38.850097 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:38.850105 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:38.850121 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:38.850379 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:38.850677 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:38.850680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:38.850681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:38.850818 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:38.851224 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:38.851232 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:38.851472 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 10:42:38.851685 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 10:42:38.851995 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":137,"threads_started":1}
I20260924 10:42:38.854435 17924 heartbeater.cc:347] Connected to a master server at 127.2.8.62:45591
I20260924 10:42:38.854689 17924 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:38.855175 17924 heartbeater.cc:511] Master 127.2.8.62:45591 requested a full tablet report, sending...
I20260924 10:42:38.856190 17465 ts_manager.cc:194] Registered new tserver with Master: 91ba2dee83384729a778675c1c0302f2 (127.2.8.3:36885)
I20260924 10:42:38.856753 17465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:58455
I20260924 10:42:38.866003 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:42:38.875460 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:38.867921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.2.8.62:45591)
0924 10:42:38.868044 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:38.868048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:38.868222 (+ 174us) server_negotiation.cc:407] Connection header received
0924 10:42:38.868307 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:38.868310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:38.868347 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:38.868409 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:38.869263 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.869960 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.870872 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.871078 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.871165 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:38.872596 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:38.872599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:38.872601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:38.872623 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:38.874106 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:38.874496 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:38.874499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:38.874500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:38.874574 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:38.874799 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:38.874801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:38.874803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:38.875001 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:38.875110 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:38.875257 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 10:42:38.875346 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 10:42:38.878159 17465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:42:38.881244 17465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:42:38.895076 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:38.889406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48042 (local address 127.2.8.2:44075)
0924 10:42:38.889767 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:38.889778 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:38.890244 (+ 466us) server_negotiation.cc:407] Connection header received
0924 10:42:38.890328 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:38.890334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:38.890554 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:38.890726 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:38.891017 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.892065 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.893598 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.894414 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.894517 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:38.894667 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:38.894841 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 10:42:38.894926 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":129,"threads_started":1}
I20260924 10:42:38.895109 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:38.889255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:44075 (local address 127.0.0.1:48042)
0924 10:42:38.889881 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:38.889930 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:38.890084 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:38.890715 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:38.890720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:38.890745 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:38.890883 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:38.890891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.892197 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.892201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:38.893478 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:38.893487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.894531 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.894535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:38.894643 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:38.894684 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 10:42:38.894802 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 10:42:38.894920 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":26,"thread_start_us":87,"threads_started":1}
I20260924 10:42:38.895639 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:38.889990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:36885 (local address 127.0.0.1:35722)
0924 10:42:38.891005 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:38.891017 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:38.891122 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:38.891655 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:38.891658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:38.891678 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:38.891766 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:38.891772 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.892930 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.892934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:38.894108 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:38.894118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.895239 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.895243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:38.895339 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:38.895366 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 10:42:38.895439 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 10:42:38.895506 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":3,"thread_start_us":106,"threads_started":1}
I20260924 10:42:38.895692 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:38.890166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35722 (local address 127.2.8.3:36885)
0924 10:42:38.890589 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:38.890601 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:38.891122 (+ 521us) server_negotiation.cc:407] Connection header received
0924 10:42:38.891220 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:38.891224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:38.891367 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:38.891490 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:38.891884 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.892789 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.894233 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.894952 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.895048 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:38.895173 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:38.895468 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 10:42:38.895558 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":121,"threads_started":1}
I20260924 10:42:38.897501 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:38.889281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:34223 (local address 127.0.0.1:47890)
0924 10:42:38.890709 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:38.890722 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:38.890799 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:38.891468 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:38.891471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:38.891489 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:38.891583 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:38.891588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.892865 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.892868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:38.895155 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:38.895169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.896949 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.896952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:38.897207 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:38.897234 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 10:42:38.897310 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 10:42:38.897379 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1365,"mutex_wait_us":1148,"thread_start_us":103,"threads_started":1}
I20260924 10:42:38.897559 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:38.890434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47890 (local address 127.2.8.1:34223)
0924 10:42:38.890993 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:38.891010 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:38.891029 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:42:38.891136 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:38.891142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:38.891318 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:38.891476 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:38.891685 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.892708 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.895311 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:38.896818 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:38.896951 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:38.897258 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:38.897337 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 10:42:38.897426 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":74,"threads_started":1}
I20260924 10:42:38.898545 17724 tablet_service.cc:1467] Processing CreateTablet for tablet 7c9384cc0aaa408ba62f994e4a7fd703 (DEFAULT_TABLE table=test-table [id=563ab1ed34204446bd80b2d3d1c89fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:42:38.898545 17858 tablet_service.cc:1467] Processing CreateTablet for tablet 7c9384cc0aaa408ba62f994e4a7fd703 (DEFAULT_TABLE table=test-table [id=563ab1ed34204446bd80b2d3d1c89fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:42:38.899705 17858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c9384cc0aaa408ba62f994e4a7fd703. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:38.899701 17724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c9384cc0aaa408ba62f994e4a7fd703. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:38.900322 17590 tablet_service.cc:1467] Processing CreateTablet for tablet 7c9384cc0aaa408ba62f994e4a7fd703 (DEFAULT_TABLE table=test-table [id=563ab1ed34204446bd80b2d3d1c89fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:42:38.901311 17590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c9384cc0aaa408ba62f994e4a7fd703. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:42:38.907013 17941 tablet_bootstrap.cc:492] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: Bootstrap starting.
I20260924 10:42:38.907644 17942 tablet_bootstrap.cc:492] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7: Bootstrap starting.
I20260924 10:42:38.909294 17941 tablet_bootstrap.cc:654] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:38.909310 17943 tablet_bootstrap.cc:492] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2: Bootstrap starting.
I20260924 10:42:38.909574 17942 tablet_bootstrap.cc:654] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:38.910737 17941 log.cc:824] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:38.910791 17942 log.cc:824] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:38.911365 17943 tablet_bootstrap.cc:654] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2: Neither blocks nor log segments found. Creating new log.
I20260924 10:42:38.912350 17943 log.cc:824] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:38.912554 17941 tablet_bootstrap.cc:492] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: No bootstrap required, opened a new log
I20260924 10:42:38.912763 17941 ts_tablet_manager.cc:1397] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:42:38.914021 17943 tablet_bootstrap.cc:492] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2: No bootstrap required, opened a new log
I20260924 10:42:38.914207 17943 ts_tablet_manager.cc:1397] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:42:38.915752 17941 raft_consensus.cc:348] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } }
I20260924 10:42:38.915985 17941 raft_consensus.cc:374] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:38.916041 17941 raft_consensus.cc:729] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28a7ed37a6f41738f3d48228a072c2d, State: Initialized, Role: FOLLOWER
I20260924 10:42:38.916479 17941 consensus_queue.cc:260] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } }
I20260924 10:42:38.917418 17942 tablet_bootstrap.cc:492] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7: No bootstrap required, opened a new log
I20260924 10:42:38.917308 17943 raft_consensus.cc:348] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } }
I20260924 10:42:38.917507 17941 ts_tablet_manager.cc:1428] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:42:38.917550 17943 raft_consensus.cc:374] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:38.917598 17943 raft_consensus.cc:729] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91ba2dee83384729a778675c1c0302f2, State: Initialized, Role: FOLLOWER
I20260924 10:42:38.917603 17942 ts_tablet_manager.cc:1397] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260924 10:42:38.917867 17656 heartbeater.cc:503] Master 127.2.8.62:45591 was elected leader, sending a full tablet report...
I20260924 10:42:38.918042 17943 consensus_queue.cc:260] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } }
I20260924 10:42:38.918679 17924 heartbeater.cc:503] Master 127.2.8.62:45591 was elected leader, sending a full tablet report...
I20260924 10:42:38.918962 17943 ts_tablet_manager.cc:1428] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:42:38.921365 17942 raft_consensus.cc:348] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } }
I20260924 10:42:38.921648 17942 raft_consensus.cc:374] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:42:38.921710 17942 raft_consensus.cc:729] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a23cf3ba12b40eb949ba189063e3fd7, State: Initialized, Role: FOLLOWER
I20260924 10:42:38.922152 17942 consensus_queue.cc:260] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } }
I20260924 10:42:38.922761 17790 heartbeater.cc:503] Master 127.2.8.62:45591 was elected leader, sending a full tablet report...
I20260924 10:42:38.923101 17942 ts_tablet_manager.cc:1428] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:42:39.035652 17948 raft_consensus.cc:482] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:42:39.035854 17948 raft_consensus.cc:504] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } }
I20260924 10:42:39.037062 17948 leader_election.cc:288] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a23cf3ba12b40eb949ba189063e3fd7 (127.2.8.2:44075), e28a7ed37a6f41738f3d48228a072c2d (127.2.8.1:34223)
I20260924 10:42:39.041016 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:39.037765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:43131 (local address 127.2.8.1:34223)
0924 10:42:39.037918 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:39.037924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:39.037940 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:42:39.037999 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:39.038001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:39.038050 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:39.038132 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:39.038436 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:39.038936 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:39.040011 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:39.040412 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:39.040528 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:39.040598 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:39.040807 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 10:42:39.040870 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:42:39.041024 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:39.037331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:34223 (local address 127.2.8.3:43131)
0924 10:42:39.037448 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:39.037464 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:39.037557 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:39.038165 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:39.038167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:39.038182 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:39.038326 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:39.038331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:39.039047 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:39.039054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:39.039886 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:39.039895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:39.040528 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:39.040532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:39.040670 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:39.040704 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:42:39.040792 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 10:42:39.040861 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":63,"spinlock_wait_cycles":6016}
I20260924 10:42:39.041028 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:39.037443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:38853 (local address 127.2.8.2:44075)
0924 10:42:39.037574 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:39.037578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:39.037921 (+ 343us) server_negotiation.cc:407] Connection header received
0924 10:42:39.038012 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:39.038015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:39.038079 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:39.038171 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:42:39.038481 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:39.038959 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:39.040021 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:39.040433 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:39.040548 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:39.040619 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:39.040825 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 10:42:39.040888 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:42:39.041062 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:39.037331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:44075 (local address 127.2.8.3:38853)
0924 10:42:39.037778 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:39.037819 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:39.037917 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:39.038185 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:39.038188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:39.038227 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:42:39.038365 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:39.038370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:39.039074 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:39.039078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:39.039901 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:39.039908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:39.040551 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:39.040554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:39.040707 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:39.040724 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:42:39.040811 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 10:42:39.040878 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":185,"spinlock_wait_cycles":1280,"thread_start_us":157,"threads_started":1}
I20260924 10:42:39.041643 17744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c9384cc0aaa408ba62f994e4a7fd703" candidate_uuid: "91ba2dee83384729a778675c1c0302f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" is_pre_election: true
I20260924 10:42:39.041669 17610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c9384cc0aaa408ba62f994e4a7fd703" candidate_uuid: "91ba2dee83384729a778675c1c0302f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28a7ed37a6f41738f3d48228a072c2d" is_pre_election: true
I20260924 10:42:39.041929 17744 raft_consensus.cc:2454] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91ba2dee83384729a778675c1c0302f2 in term 0.
I20260924 10:42:39.041945 17610 raft_consensus.cc:2454] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91ba2dee83384729a778675c1c0302f2 in term 0.
I20260924 10:42:39.042414 17811 leader_election.cc:302] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a23cf3ba12b40eb949ba189063e3fd7, 91ba2dee83384729a778675c1c0302f2; no voters:
I20260924 10:42:39.042703 17948 raft_consensus.cc:2781] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:42:39.042788 17948 raft_consensus.cc:482] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:42:39.042820 17948 raft_consensus.cc:3037] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:39.043854 17948 raft_consensus.cc:504] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } }
I20260924 10:42:39.044226 17948 leader_election.cc:288] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [CANDIDATE]: Term 1 election: Requested vote from peers 1a23cf3ba12b40eb949ba189063e3fd7 (127.2.8.2:44075), e28a7ed37a6f41738f3d48228a072c2d (127.2.8.1:34223)
I20260924 10:42:39.044512 17744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c9384cc0aaa408ba62f994e4a7fd703" candidate_uuid: "91ba2dee83384729a778675c1c0302f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a23cf3ba12b40eb949ba189063e3fd7"
I20260924 10:42:39.044639 17744 raft_consensus.cc:3037] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:39.044727 17610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c9384cc0aaa408ba62f994e4a7fd703" candidate_uuid: "91ba2dee83384729a778675c1c0302f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28a7ed37a6f41738f3d48228a072c2d"
I20260924 10:42:39.044907 17610 raft_consensus.cc:3037] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:42:39.045742 17744 raft_consensus.cc:2454] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91ba2dee83384729a778675c1c0302f2 in term 1.
I20260924 10:42:39.046026 17811 leader_election.cc:302] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a23cf3ba12b40eb949ba189063e3fd7, 91ba2dee83384729a778675c1c0302f2; no voters:
I20260924 10:42:39.046089 17610 raft_consensus.cc:2454] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91ba2dee83384729a778675c1c0302f2 in term 1.
I20260924 10:42:39.046202 17948 raft_consensus.cc:2781] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:42:39.046434 17948 raft_consensus.cc:686] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [term 1 LEADER]: Becoming Leader. State: Replica: 91ba2dee83384729a778675c1c0302f2, State: Running, Role: LEADER
I20260924 10:42:39.046753 17948 consensus_queue.cc:237] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } }
I20260924 10:42:39.049886 17465 catalog_manager.cc:5495] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91ba2dee83384729a778675c1c0302f2 (127.2.8.3). New cstate: current_term: 1 leader_uuid: "91ba2dee83384729a778675c1c0302f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } health_report { overall_health: HEALTHY } } }
I20260924 10:42:39.082772 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:39.079311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.2.8.3:36885)
0924 10:42:39.079436 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:39.079440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:39.079452 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:42:39.079524 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:39.079526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:39.079585 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:39.079673 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:42:39.079934 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:39.080434 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:39.081380 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:39.081588 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:39.081660 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:39.081942 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:42:39.082340 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:42:39.082431 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:39.082623 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 10:42:39.082669 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 10:42:39.088769 17925 tablet.cc:2369] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:42:39.089119 17610 raft_consensus.cc:1261] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d [term 1 FOLLOWER]: Refusing update from remote peer 91ba2dee83384729a778675c1c0302f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:42:39.089167 17744 raft_consensus.cc:1261] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 1 FOLLOWER]: Refusing update from remote peer 91ba2dee83384729a778675c1c0302f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:42:39.089879 17951 consensus_queue.cc:1035] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:42:39.090095 17948 consensus_queue.cc:1035] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:42:39.098294 17954 mvcc.cc:205] Tried to move back new op lower bound from 7332849906021343232 to 7332849905857691648. Current Snapshot: MvccSnapshot[applied={T|T < 7332849906021343232}]
I20260924 10:42:39.098829 17955 mvcc.cc:205] Tried to move back new op lower bound from 7332849906021343232 to 7332849905857691648. Current Snapshot: MvccSnapshot[applied={T|T < 7332849906021343232}]
I20260924 10:42:39.105931 17465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:42:39.106099 17465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:42:39.108565 17465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Sending DeleteTablet for 3 replicas of tablet 7c9384cc0aaa408ba62f994e4a7fd703
I20260924 10:42:39.109368 17724 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c9384cc0aaa408ba62f994e4a7fd703 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:42:39 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:48042
I20260924 10:42:39.109637 17858 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c9384cc0aaa408ba62f994e4a7fd703 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:42:39 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:35722
I20260924 10:42:39.109956 17965 tablet_replica.cc:330] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7: stopping tablet replica
I20260924 10:42:39.109928 17590 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c9384cc0aaa408ba62f994e4a7fd703 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:42:39 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:47890
I20260924 10:42:39.110280 17965 raft_consensus.cc:2229] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:42:39.110484 17967 tablet_replica.cc:330] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: stopping tablet replica
I20260924 10:42:39.110509 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 17527
I20260924 10:42:39.110590 17965 raft_consensus.cc:2258] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:42:39.112560 17965 ts_tablet_manager.cc:1905] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:42:39.115527 17965 ts_tablet_manager.cc:1918] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:42:39.115625 17965 log.cc:1196] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/7c9384cc0aaa408ba62f994e4a7fd703
I20260924 10:42:39.115932 17965 ts_tablet_manager.cc:1939] T 7c9384cc0aaa408ba62f994e4a7fd703 P 1a23cf3ba12b40eb949ba189063e3fd7: Deleting consensus metadata
I20260924 10:42:39.117177 17449 catalog_manager.cc:4841] TS 1a23cf3ba12b40eb949ba189063e3fd7 (127.2.8.2:44075): tablet 7c9384cc0aaa408ba62f994e4a7fd703 (table test-table [id=563ab1ed34204446bd80b2d3d1c89fcd]) successfully deleted
W20260924 10:42:39.118144 17451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.8.1:34223 (error 108)
I20260924 10:42:39.118429 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 17661
I20260924 10:42:39.118498 17966 tablet_replica.cc:330] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2: stopping tablet replica
I20260924 10:42:39.120798 17966 raft_consensus.cc:2229] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:42:39.121280 17966 raft_consensus.cc:2258] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:42:39.121843 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:39.121440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:34223 (local address 127.0.0.1:47896)
0924 10:42:39.121646 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:39.121733 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260924 10:42:39.122030 17451 catalog_manager.cc:4584] TS e28a7ed37a6f41738f3d48228a072c2d (127.2.8.1:34223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c9384cc0aaa408ba62f994e4a7fd703: Network error: Client connection negotiation failed: client connection to 127.2.8.1:34223: connect: Connection refused (error 111)
I20260924 10:42:39.123754 17966 ts_tablet_manager.cc:1905] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:42:39.125918 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 17795
I20260924 10:42:39.130904 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 17433
I20260924 10:42:59.136590 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:45591
--webserver_interface=127.2.8.62
--webserver_port=46579
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:45591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:42:59.249864 17969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:59.250118 17969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:59.250207 17969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:59.254177 17969 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 10:42:59.254230 17969 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 10:42:59.254245 17969 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 10:42:59.254259 17969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:42:59.254329 17969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:59.254369 17969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:59.254410 17969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:42:59.254456 17969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:42:59.259778 17969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:45591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:45591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=46579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:59.260867 17969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:59.261902 17969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:59.278391 17978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:59.278434 17976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:59.278391 17975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:59.278707 17969 server_base.cc:1034] running on GCE node
I20260924 10:42:59.279392 17969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:59.280634 17969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:59.281819 17969 hybrid_clock.cc:648] HybridClock initialized: now 1790246579281809 us; error 31 us; skew 500 ppm
Sep 24 10:42:59 dist-test-slave-j38h krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:59.291368 17969 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:42:59.292352 17969 webserver.cc:460] Webserver started at http://127.2.8.62:46579/ using document root <none> and password file <none>
I20260924 10:42:59.293068 17969 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:59.293171 17969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:59.297116 17969 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 10:42:59.299185 17985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:59.300454 17969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:42:59.300616 17969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "47797fad5d7e4511b1040bcbaa4b7048"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "5bac52b6c910de4bc9ca86c2bd38229d"
server_key_iv: "a0e8cf6d8d60a5ada6de1b2fb3c658e5"
server_key_version: "encryptionkey@0"
I20260924 10:42:59.301256 17969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:59.339444 17969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:59.343075 17969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:59.343294 17969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:59.350642 17969 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:45591
I20260924 10:42:59.350931 18037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:45591 every 8 connection(s)
I20260924 10:42:59.351779 17969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:42:59.353773 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 17969
I20260924 10:42:59.354288 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:34223
--local_ip_for_outbound_sockets=127.2.8.1
--tserver_master_addrs=127.2.8.62:45591
--webserver_port=44557
--webserver_interface=127.2.8.1
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:42:59.357263 18038 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:42:59.359901 18038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Bootstrap starting.
I20260924 10:42:59.383302 18038 log.cc:824] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Log is configured to *not* fsync() on all Append() calls
I20260924 10:42:59.393863 18038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:42:59.394290 18038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Bootstrap complete.
I20260924 10:42:59.399863 18038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } }
I20260924 10:42:59.400396 18038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47797fad5d7e4511b1040bcbaa4b7048, State: Initialized, Role: FOLLOWER
I20260924 10:42:59.400951 18038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } }
I20260924 10:42:59.401134 18038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:42:59.401211 18038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:42:59.401309 18038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:42:59.403698 18038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } }
I20260924 10:42:59.404072 18038 leader_election.cc:302] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47797fad5d7e4511b1040bcbaa4b7048; no voters:
I20260924 10:42:59.404394 18038 leader_election.cc:288] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 10:42:59.404507 18042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:42:59.404831 18042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 2 LEADER]: Becoming Leader. State: Replica: 47797fad5d7e4511b1040bcbaa4b7048, State: Running, Role: LEADER
I20260924 10:42:59.405224 18042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } }
I20260924 10:42:59.405687 18038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:42:59.406208 18043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "47797fad5d7e4511b1040bcbaa4b7048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } } }
I20260924 10:42:59.406213 18042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47797fad5d7e4511b1040bcbaa4b7048. Latest consensus state: current_term: 2 leader_uuid: "47797fad5d7e4511b1040bcbaa4b7048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } } }
I20260924 10:42:59.406476 18042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:59.406508 18043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [sys.catalog]: This master's current role is: LEADER
I20260924 10:42:59.406854 18049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:42:59.412011 18049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:42:59.412834 18049 catalog_manager.cc:1259] Loaded cluster ID: fc2ea8eb88ae49978761aca9042f1b9e
I20260924 10:42:59.412912 18049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:42:59.414745 18049 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:42:59.415335 18049 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Loaded TSK: 0
I20260924 10:42:59.426867 18049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Generated new TSK 1
I20260924 10:42:59.427124 18049 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:42:59.501410 18040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:59.501652 18040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:59.501693 18040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:59.505997 18040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:59.506074 18040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:59.506167 18040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:42:59.511577 18040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:34223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=44557
--tserver_master_addrs=127.2.8.62:45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:59.512564 18040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:59.513604 18040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:59.522868 18068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:59.523000 18066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:59.522868 18065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:59.523254 18040 server_base.cc:1034] running on GCE node
I20260924 10:42:59.523654 18040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:59.524276 18040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:59.525455 18040 hybrid_clock.cc:648] HybridClock initialized: now 1790246579525427 us; error 43 us; skew 500 ppm
Sep 24 10:42:59 dist-test-slave-j38h krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:59.528182 18040 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:42:59.529150 18040 webserver.cc:460] Webserver started at http://127.2.8.1:44557/ using document root <none> and password file <none>
I20260924 10:42:59.529774 18040 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:59.529889 18040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:59.533754 18040 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:42:59.536041 18075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:59.537382 18040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:42:59.537525 18040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "e28a7ed37a6f41738f3d48228a072c2d"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "5741d069547885c52b100e3e17e97476"
server_key_iv: "cb9d77d7189dcceff5e9131d9963baf0"
server_key_version: "encryptionkey@0"
I20260924 10:42:59.538067 18040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:59.561362 18040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:59.564733 18040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:59.564955 18040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:59.565585 18040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:59.566749 18082 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 10:42:59.570829 18040 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:42:59.570889 18040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260924 10:42:59.570936 18040 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:42:59.572628 18040 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:42:59.572681 18040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 10:42:59.572942 18082 tablet_bootstrap.cc:492] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: Bootstrap starting.
I20260924 10:42:59.583519 18040 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:34223
I20260924 10:42:59.583612 18189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:34223 every 8 connection(s)
I20260924 10:42:59.584726 18040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:42:59.591984 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 18040
I20260924 10:42:59.592490 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:44075
--local_ip_for_outbound_sockets=127.2.8.2
--tserver_master_addrs=127.2.8.62:45591
--webserver_port=39115
--webserver_interface=127.2.8.2
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:42:59.593468 18082 log.cc:824] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:42:59 dist-test-slave-j38h krb5kdc[17420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:59.601264 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:59.586916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:50743 (local address 127.2.8.62:45591)
0924 10:42:59.587281 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:59.587300 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:59.587628 (+ 328us) server_negotiation.cc:407] Connection header received
0924 10:42:59.588732 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:59.588742 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:59.589105 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:59.589426 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:59.590484 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.591455 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.592746 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.593311 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.593508 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:59.596321 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:59.596327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:59.596338 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:59.596365 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:59.598131 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:59.598684 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:59.598690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:59.598702 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:59.598821 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:59.599168 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:59.599173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:59.599175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:59.599499 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:59.599632 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:59.600172 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 10:42:59.600430 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":133,"threads_started":1}
I20260924 10:42:59.601274 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:59.586937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:50743)
0924 10:42:59.587448 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:59.587487 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:59.588317 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:59.589463 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:59.589470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:59.589881 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:59.590315 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:59.590328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.591615 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.591618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:59.592592 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:59.592606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.593431 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.593468 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:59.593712 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:59.594517 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:59.594541 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:59.596128 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:59.598286 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:59.598291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:59.598319 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:59.598570 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:59.598931 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:59.598934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:59.598936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:59.599073 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:59.599621 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:59.599627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:59.599866 (+ 239us) client_negotiation.cc:769] Sending connection context
0924 10:42:59.600052 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 10:42:59.600378 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":146,"threads_started":1}
I20260924 10:42:59.602571 18082 tablet_bootstrap.cc:492] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:42:59.603029 18082 tablet_bootstrap.cc:492] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: Bootstrap complete.
I20260924 10:42:59.604095 18190 heartbeater.cc:347] Connected to a master server at 127.2.8.62:45591
I20260924 10:42:59.604322 18190 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:59.604492 18082 ts_tablet_manager.cc:1397] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.012s
I20260924 10:42:59.604836 18190 heartbeater.cc:511] Master 127.2.8.62:45591 requested a full tablet report, sending...
I20260924 10:42:59.607321 18002 ts_manager.cc:194] Registered new tserver with Master: e28a7ed37a6f41738f3d48228a072c2d (127.2.8.1:34223)
I20260924 10:42:59.607476 18082 raft_consensus.cc:348] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } }
I20260924 10:42:59.608026 18082 raft_consensus.cc:729] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28a7ed37a6f41738f3d48228a072c2d, State: Initialized, Role: FOLLOWER
I20260924 10:42:59.608486 18082 consensus_queue.cc:260] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } }
I20260924 10:42:59.609455 18082 ts_tablet_manager.cc:1428] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260924 10:42:59.610908 18002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:50743
I20260924 10:42:59.613325 18190 heartbeater.cc:503] Master 127.2.8.62:45591 was elected leader, sending a full tablet report...
Sep 24 10:42:59 dist-test-slave-j38h krb5kdc[17420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for kudu/127.2.8.1@KRBTEST.COM
I20260924 10:42:59.621426 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:59.610252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:34223 (local address 127.0.0.1:40850)
0924 10:42:59.610828 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:59.610878 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:59.611010 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:59.612076 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:59.612082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:59.612551 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:59.612691 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:59.612699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.613982 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.613984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:59.614585 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:59.614593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.615377 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.615392 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:59.615474 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:59.616078 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:59.616089 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:59.617654 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:59.619622 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:59.619630 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:59.619634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:59.619853 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:59.620191 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:59.620196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:59.620202 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:59.620305 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:59.620838 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:59.620843 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:59.620950 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 10:42:59.621133 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 10:42:59.621286 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":101,"threads_started":1}
I20260924 10:42:59.621539 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:59.610342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.2.8.1:34223)
0924 10:42:59.611664 (+ 1322us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:59.611673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:59.611690 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:42:59.611737 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:59.611741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:59.611924 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:59.612051 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:59.612827 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.613858 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.614721 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.615247 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.615352 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:59.617784 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:59.617794 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:59.617805 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:59.617836 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:59.619503 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:59.619966 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:59.619972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:59.619974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:59.620060 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:59.620400 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:59.620404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:59.620405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:59.620669 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:59.620832 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:59.621229 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 10:42:59.621383 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1235,"thread_start_us":93,"threads_started":1}
I20260924 10:42:59.622473 18124 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c9384cc0aaa408ba62f994e4a7fd703 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:42:39 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:40850
I20260924 10:42:59.623275 18199 tablet_replica.cc:330] stopping tablet replica
I20260924 10:42:59.623569 18199 raft_consensus.cc:2229] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:42:59.623817 18199 raft_consensus.cc:2258] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:42:59.624835 18199 ts_tablet_manager.cc:1905] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:42:59.629098 18199 ts_tablet_manager.cc:1918] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:42:59.629191 18199 log.cc:1196] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal/wals/7c9384cc0aaa408ba62f994e4a7fd703
I20260924 10:42:59.629451 18199 ts_tablet_manager.cc:1939] T 7c9384cc0aaa408ba62f994e4a7fd703 P e28a7ed37a6f41738f3d48228a072c2d: Deleting consensus metadata
I20260924 10:42:59.631351 17988 catalog_manager.cc:4841] TS e28a7ed37a6f41738f3d48228a072c2d (127.2.8.1:34223): tablet 7c9384cc0aaa408ba62f994e4a7fd703 (table test-table [id=563ab1ed34204446bd80b2d3d1c89fcd]) successfully deleted
W20260924 10:42:59.712200 18194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:59.712483 18194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:59.712527 18194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:59.716619 18194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:59.716683 18194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:59.716774 18194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:42:59.721902 18194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:44075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=39115
--tserver_master_addrs=127.2.8.62:45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:59.722899 18194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:59.723899 18194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:59.733165 18204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:59.733165 18205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:59.733211 18207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:59.733242 18194 server_base.cc:1034] running on GCE node
I20260924 10:42:59.733819 18194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:59.734414 18194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:59.735620 18194 hybrid_clock.cc:648] HybridClock initialized: now 1790246579735591 us; error 46 us; skew 500 ppm
Sep 24 10:42:59 dist-test-slave-j38h krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:59.738325 18194 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:42:59.739171 18194 webserver.cc:460] Webserver started at http://127.2.8.2:39115/ using document root <none> and password file <none>
I20260924 10:42:59.739724 18194 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:59.739792 18194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:59.743616 18194 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:42:59.745702 18214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:59.746979 18194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:42:59.747078 18194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "1a23cf3ba12b40eb949ba189063e3fd7"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "3ad746a293de0c9bb70a4206bd1604f8"
server_key_iv: "ccac7ddd65b64fcc2337a3e05d95e91d"
server_key_version: "encryptionkey@0"
I20260924 10:42:59.747648 18194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:59.760142 18194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:59.763127 18194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:59.763276 18194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:59.763783 18194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:59.764539 18194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:42:59.764583 18194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:59.764621 18194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:59.764637 18194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:59.774019 18194 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:44075
I20260924 10:42:59.774046 18327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:44075 every 8 connection(s)
I20260924 10:42:59.774996 18194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:42:59.780160 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 18194
I20260924 10:42:59.780615 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:36885
--local_ip_for_outbound_sockets=127.2.8.3
--tserver_master_addrs=127.2.8.62:45591
--webserver_port=45119
--webserver_interface=127.2.8.3
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:42:59 dist-test-slave-j38h krb5kdc[17420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:59.789417 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:59.776779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:50127 (local address 127.2.8.62:45591)
0924 10:42:59.776916 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:59.776919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:59.777980 (+ 1061us) server_negotiation.cc:407] Connection header received
0924 10:42:59.778892 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:59.778896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:59.778961 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:59.779075 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:59.780263 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.780907 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.782068 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.782306 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.782416 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:59.785257 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:59.785263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:59.785269 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:59.785308 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:59.787141 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:59.787683 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:59.787689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:59.787692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:59.787793 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:59.788143 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:59.788149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:59.788152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:59.788413 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:59.788542 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:59.789020 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 10:42:59.789181 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:42:59.790114 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:59.777180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:50127)
0924 10:42:59.777821 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:59.777860 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:59.778675 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:59.779253 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:59.779260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:59.779625 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:59.780110 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:59.780124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.781049 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.781053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:59.781932 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:59.781941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.782414 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.782468 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:59.782640 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:59.783270 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:59.783295 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:59.785080 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:59.787276 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:59.787281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:59.787293 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:59.787564 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:59.787892 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:59.787895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:59.787896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:59.788034 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:59.788527 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:59.788531 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:59.788752 (+ 221us) client_negotiation.cc:769] Sending connection context
0924 10:42:59.788896 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 10:42:59.789227 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":108,"threads_started":1}
I20260924 10:42:59.791424 18328 heartbeater.cc:347] Connected to a master server at 127.2.8.62:45591
I20260924 10:42:59.791682 18328 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:59.792169 18328 heartbeater.cc:511] Master 127.2.8.62:45591 requested a full tablet report, sending...
I20260924 10:42:59.793288 18002 ts_manager.cc:194] Registered new tserver with Master: 1a23cf3ba12b40eb949ba189063e3fd7 (127.2.8.2:44075)
I20260924 10:42:59.793857 18002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:50127
W20260924 10:42:59.893087 18331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:42:59.893368 18331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:42:59.893468 18331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:42:59.897473 18331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:42:59.897565 18331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:42:59.897660 18331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:42:59.902719 18331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:36885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=45119
--tserver_master_addrs=127.2.8.62:45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:42:59.903708 18331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:42:59.904799 18331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:42:59.914567 18339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:59.914644 18336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:42:59.914587 18337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:42:59.914988 18331 server_base.cc:1034] running on GCE node
I20260924 10:42:59.915303 18331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:42:59.915876 18331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:42:59.917058 18331 hybrid_clock.cc:648] HybridClock initialized: now 1790246579917029 us; error 43 us; skew 500 ppm
Sep 24 10:42:59 dist-test-slave-j38h krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:42:59.919574 18331 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:42:59.920428 18331 webserver.cc:460] Webserver started at http://127.2.8.3:45119/ using document root <none> and password file <none>
I20260924 10:42:59.920960 18331 fs_manager.cc:359] Metadata directory not provided
I20260924 10:42:59.921123 18331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:42:59.924669 18331 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:42:59.926900 18346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:59.928090 18331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:42:59.928210 18331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "91ba2dee83384729a778675c1c0302f2"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "0dc19bd34d34f25f3b7d2be9c25bd5d9"
server_key_iv: "225f9ea178b2cb44db1c9adca1303438"
server_key_version: "encryptionkey@0"
I20260924 10:42:59.928776 18331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:42:59.952001 18331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:42:59.954835 18331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:42:59.954999 18331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:42:59.955518 18331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:42:59.956568 18353 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 10:42:59.960394 18353 ts_tablet_manager.cc:1852] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 10:42:59.960786 18353 ts_tablet_manager.cc:1905] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:42:59.964751 18353 ts_tablet_manager.cc:1918] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:42:59.964862 18353 log.cc:1196] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/7c9384cc0aaa408ba62f994e4a7fd703
I20260924 10:42:59.965260 18353 ts_tablet_manager.cc:1939] T 7c9384cc0aaa408ba62f994e4a7fd703 P 91ba2dee83384729a778675c1c0302f2: Deleting consensus metadata
I20260924 10:42:59.965592 18331 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 10:42:59.965674 18331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260924 10:42:59.965780 18331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:42:59.965863 18331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:42:59.974969 18331 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:36885
I20260924 10:42:59.975004 18460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:36885 every 8 connection(s)
I20260924 10:42:59.975802 18331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:42:59.977892 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 18331
Sep 24 10:42:59 dist-test-slave-j38h krb5kdc[17420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246579, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:42:59.987648 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:59.978540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.2.8.62:45591)
0924 10:42:59.978761 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:59.978765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:59.979053 (+ 288us) server_negotiation.cc:407] Connection header received
0924 10:42:59.979185 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:59.979187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:59.979231 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:59.979312 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:59.980299 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.980903 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.981919 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.982220 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.982293 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:59.983870 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:59.983873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:59.983876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:59.983903 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:59.985516 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:59.986119 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:59.986126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:59.986129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:59.986245 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:59.986643 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:59.986645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:59.986646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:59.986897 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:59.987137 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:59.987314 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 10:42:59.987433 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
I20260924 10:42:59.990046 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:59.977637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:51345 (local address 127.2.8.62:45591)
0924 10:42:59.977810 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 10:42:59.977816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:42:59.978878 (+ 1062us) server_negotiation.cc:407] Connection header received
0924 10:42:59.980131 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:42:59.980133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:42:59.980168 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:42:59.980222 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:42:59.981484 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.982005 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.983629 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.983906 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.984016 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:59.986276 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:42:59.986282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:42:59.986286 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:42:59.986313 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:42:59.987977 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:59.988611 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:59.988616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:59.988617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:59.988701 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:42:59.989094 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:42:59.989096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:42:59.989098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:42:59.989245 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:42:59.989320 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:42:59.989852 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 10:42:59.989942 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:42:59.990851 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:42:59.978065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:51345)
0924 10:42:59.978697 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0924 10:42:59.978754 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 10:42:59.979904 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:42:59.980392 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:42:59.980400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:42:59.980717 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:42:59.981333 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:59.981351 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.982117 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.982120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:59.983396 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:42:59.983409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:42:59.984010 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:42:59.984052 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:42:59.984183 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:42:59.984709 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:42:59.984724 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:42:59.986135 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:42:59.988084 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:59.988089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:59.988099 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:59.988489 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:59.988797 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:42:59.988799 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:42:59.988801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:42:59.989006 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:42:59.989349 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:42:59.989354 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:42:59.989637 (+ 283us) client_negotiation.cc:769] Sending connection context
0924 10:42:59.989782 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 10:42:59.990024 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":161,"threads_started":1}
I20260924 10:42:59.991879 18461 heartbeater.cc:347] Connected to a master server at 127.2.8.62:45591
I20260924 10:42:59.992108 18461 heartbeater.cc:464] Registering TS with master...
I20260924 10:42:59.992573 18461 heartbeater.cc:511] Master 127.2.8.62:45591 requested a full tablet report, sending...
I20260924 10:42:59.993652 18002 ts_manager.cc:194] Registered new tserver with Master: 91ba2dee83384729a778675c1c0302f2 (127.2.8.3:36885)
I20260924 10:42:59.994202 18002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:51345
I20260924 10:43:00.001266 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:43:00.005520 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:00.001913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.2.8.62:45591)
0924 10:43:00.002042 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:00.002045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:00.002261 (+ 216us) server_negotiation.cc:407] Connection header received
0924 10:43:00.002412 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:00.002414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:00.002499 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:00.002583 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:43:00.002787 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.003288 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.004232 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.004477 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.004569 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:00.004920 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:43:00.005105 (+ 185us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 10:43:00.005425 (+ 320us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58980: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 10:43:00.005573 18192 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58980: token expired
W20260924 10:43:00.005874 17928 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.8.62:45591)
I20260924 10:43:00.012619 18192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:00.006262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.2.8.62:45591)
0924 10:43:00.006361 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:00.006366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:00.006409 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:43:00.006490 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:00.006492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:00.006524 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:00.006584 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:00.007233 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.007688 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.008421 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.008623 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.008703 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:00.010016 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:00.010026 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:00.010028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:00.010048 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:00.011303 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:00.011669 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:00.011671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:00.011672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:00.011741 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:00.011950 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:00.011952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:00.011953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:00.012115 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:00.012211 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:00.012407 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 10:43:00.012499 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 10:43:00.014979 18002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:43:00.018149 18002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:43:00.025640 18124 tablet_service.cc:1467] Processing CreateTablet for tablet 54c76aae1f6747298a3a64c7b8c462b8 (DEFAULT_TABLE table=test-table [id=9d51988c211040acbc1ebc03be42fce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:00.026304 18124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c76aae1f6747298a3a64c7b8c462b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:00.028712 18082 tablet_bootstrap.cc:492] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d: Bootstrap starting.
I20260924 10:43:00.029625 18082 tablet_bootstrap.cc:654] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:00.030459 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:00.024413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:44075 (local address 127.0.0.1:57090)
0924 10:43:00.024551 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:00.024566 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:00.024640 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:00.025728 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:00.025731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:00.025762 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:00.025855 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:00.025861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.027321 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.027324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:00.028645 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:00.028656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.029915 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.029919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:00.030065 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:00.030125 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 10:43:00.030209 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 10:43:00.030302 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":68,"spinlock_wait_cycles":8576}
I20260924 10:43:00.030613 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:00.024684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:36885 (local address 127.0.0.1:50864)
0924 10:43:00.024932 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:00.024944 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:00.025045 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:00.025712 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:00.025715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:00.025731 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:00.025847 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:00.025857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.027199 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.027205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:00.028645 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:00.028656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.030213 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.030217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:00.030303 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:00.030323 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:43:00.030407 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 10:43:00.030476 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":80,"thread_start_us":99,"threads_started":1}
I20260924 10:43:00.030679 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:00.025004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.2.8.3:36885)
0924 10:43:00.025318 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:00.025327 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:00.025344 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:43:00.025404 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:00.025407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:00.025560 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:00.025721 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:00.025980 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.027091 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.029156 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.030085 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.030211 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:00.030340 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:00.030435 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 10:43:00.030516 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":128,"threads_started":1}
I20260924 10:43:00.031250 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:00.024780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.2.8.2:44075)
0924 10:43:00.025221 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:00.025234 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:00.025258 (+ 24us) server_negotiation.cc:407] Connection header received
0924 10:43:00.025337 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:00.025342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:00.025566 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:00.026010 (+ 444us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:00.026183 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.027213 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.028882 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.029780 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.029893 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:00.030039 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:00.031045 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0924 10:43:00.031142 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":78,"threads_started":1}
I20260924 10:43:00.031461 18082 tablet_bootstrap.cc:492] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d: No bootstrap required, opened a new log
I20260924 10:43:00.031558 18082 ts_tablet_manager.cc:1397] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 10:43:00.031894 18082 raft_consensus.cc:348] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:00.032019 18082 raft_consensus.cc:374] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:00.032074 18082 raft_consensus.cc:729] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28a7ed37a6f41738f3d48228a072c2d, State: Initialized, Role: FOLLOWER
I20260924 10:43:00.032212 18082 consensus_queue.cc:260] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:00.032456 18082 ts_tablet_manager.cc:1428] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 10:43:00.032119 18395 tablet_service.cc:1467] Processing CreateTablet for tablet 54c76aae1f6747298a3a64c7b8c462b8 (DEFAULT_TABLE table=test-table [id=9d51988c211040acbc1ebc03be42fce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:00.033254 18395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c76aae1f6747298a3a64c7b8c462b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:00.034273 18262 tablet_service.cc:1467] Processing CreateTablet for tablet 54c76aae1f6747298a3a64c7b8c462b8 (DEFAULT_TABLE table=test-table [id=9d51988c211040acbc1ebc03be42fce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:00.035231 18262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c76aae1f6747298a3a64c7b8c462b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:00.037792 18353 tablet_bootstrap.cc:492] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: Bootstrap starting.
I20260924 10:43:00.039616 18353 tablet_bootstrap.cc:654] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:00.040328 18353 log.cc:824] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:00.041759 18475 tablet_bootstrap.cc:492] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7: Bootstrap starting.
I20260924 10:43:00.042474 18353 tablet_bootstrap.cc:492] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: No bootstrap required, opened a new log
I20260924 10:43:00.042698 18353 ts_tablet_manager.cc:1397] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260924 10:43:00.043861 18475 tablet_bootstrap.cc:654] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:00.045110 18475 log.cc:824] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:00.046648 18353 raft_consensus.cc:348] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:00.046917 18353 raft_consensus.cc:374] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:00.046981 18353 raft_consensus.cc:729] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91ba2dee83384729a778675c1c0302f2, State: Initialized, Role: FOLLOWER
I20260924 10:43:00.047102 18475 tablet_bootstrap.cc:492] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7: No bootstrap required, opened a new log
I20260924 10:43:00.047297 18475 ts_tablet_manager.cc:1397] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:43:00.047484 18353 consensus_queue.cc:260] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:00.048163 18461 heartbeater.cc:503] Master 127.2.8.62:45591 was elected leader, sending a full tablet report...
I20260924 10:43:00.048409 18353 ts_tablet_manager.cc:1428] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:43:00.050741 18475 raft_consensus.cc:348] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:00.051005 18475 raft_consensus.cc:374] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:00.051069 18475 raft_consensus.cc:729] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a23cf3ba12b40eb949ba189063e3fd7, State: Initialized, Role: FOLLOWER
I20260924 10:43:00.051548 18475 consensus_queue.cc:260] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:00.052189 18328 heartbeater.cc:503] Master 127.2.8.62:45591 was elected leader, sending a full tablet report...
I20260924 10:43:00.052460 18475 ts_tablet_manager.cc:1428] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260924 10:43:00.085983 18191 tablet.cc:2369] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:43:00.226717 18462 tablet.cc:2369] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:43:00.257261 18477 raft_consensus.cc:482] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:43:00.257489 18477 raft_consensus.cc:504] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:00.258694 18477 leader_election.cc:288] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a23cf3ba12b40eb949ba189063e3fd7 (127.2.8.2:44075), e28a7ed37a6f41738f3d48228a072c2d (127.2.8.1:34223)
I20260924 10:43:00.262668 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:00.259001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:34223 (local address 127.2.8.3:53053)
0924 10:43:00.259116 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:00.259135 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:00.259239 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:00.259710 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:00.259713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:00.259756 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:00.259867 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:00.259872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.260625 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.260630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:00.261489 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:00.261498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.262260 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.262263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:00.262362 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:00.262383 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:43:00.262454 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 10:43:00.262532 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"spinlock_wait_cycles":6912}
I20260924 10:43:00.262669 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:00.259106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:53053 (local address 127.2.8.1:34223)
0924 10:43:00.259404 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:00.259409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:00.259428 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:43:00.259493 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:00.259496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:00.259567 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:00.259701 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:00.259979 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.260511 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.261619 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.262124 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.262250 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:00.262424 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:00.262465 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 10:43:00.262533 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":135,"threads_started":1}
I20260924 10:43:00.262970 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:00.259085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:41743 (local address 127.2.8.2:44075)
0924 10:43:00.259222 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:00.259227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:00.259699 (+ 472us) server_negotiation.cc:407] Connection header received
0924 10:43:00.259858 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:00.259861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:00.259913 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:00.259997 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:00.260256 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.260744 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.261800 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.262314 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.262416 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:00.262474 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:00.262782 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 10:43:00.262851 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:43:00.263389 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:00.259002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:44075 (local address 127.2.8.3:41743)
0924 10:43:00.259614 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:00.259624 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:00.259758 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:00.260004 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:00.260007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:00.260033 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:00.260151 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:00.260156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.260855 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.260860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:00.261680 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:00.261689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.262547 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.262552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:00.262643 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:00.262664 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:43:00.263157 (+ 493us) client_negotiation.cc:240] Negotiation successful
0924 10:43:00.263227 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":303,"spinlock_wait_cycles":20224,"thread_start_us":116,"threads_started":1}
I20260924 10:43:00.263447 18144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c76aae1f6747298a3a64c7b8c462b8" candidate_uuid: "91ba2dee83384729a778675c1c0302f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28a7ed37a6f41738f3d48228a072c2d" is_pre_election: true
I20260924 10:43:00.263795 18144 raft_consensus.cc:2454] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91ba2dee83384729a778675c1c0302f2 in term 0.
I20260924 10:43:00.264005 18282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c76aae1f6747298a3a64c7b8c462b8" candidate_uuid: "91ba2dee83384729a778675c1c0302f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" is_pre_election: true
I20260924 10:43:00.264276 18349 leader_election.cc:302] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91ba2dee83384729a778675c1c0302f2, e28a7ed37a6f41738f3d48228a072c2d; no voters:
I20260924 10:43:00.264324 18282 raft_consensus.cc:2454] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91ba2dee83384729a778675c1c0302f2 in term 0.
I20260924 10:43:00.264578 18477 raft_consensus.cc:2781] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:43:00.264701 18477 raft_consensus.cc:482] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:43:00.264766 18477 raft_consensus.cc:3037] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:00.265830 18477 raft_consensus.cc:504] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:00.266180 18477 leader_election.cc:288] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [CANDIDATE]: Term 1 election: Requested vote from peers 1a23cf3ba12b40eb949ba189063e3fd7 (127.2.8.2:44075), e28a7ed37a6f41738f3d48228a072c2d (127.2.8.1:34223)
I20260924 10:43:00.266520 18144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c76aae1f6747298a3a64c7b8c462b8" candidate_uuid: "91ba2dee83384729a778675c1c0302f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e28a7ed37a6f41738f3d48228a072c2d"
I20260924 10:43:00.266566 18282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c76aae1f6747298a3a64c7b8c462b8" candidate_uuid: "91ba2dee83384729a778675c1c0302f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a23cf3ba12b40eb949ba189063e3fd7"
I20260924 10:43:00.266654 18144 raft_consensus.cc:3037] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:00.266693 18282 raft_consensus.cc:3037] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:00.267704 18144 raft_consensus.cc:2454] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91ba2dee83384729a778675c1c0302f2 in term 1.
I20260924 10:43:00.267863 18282 raft_consensus.cc:2454] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91ba2dee83384729a778675c1c0302f2 in term 1.
I20260924 10:43:00.268025 18349 leader_election.cc:302] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91ba2dee83384729a778675c1c0302f2, e28a7ed37a6f41738f3d48228a072c2d; no voters:
I20260924 10:43:00.268241 18477 raft_consensus.cc:2781] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:00.268425 18477 raft_consensus.cc:686] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [term 1 LEADER]: Becoming Leader. State: Replica: 91ba2dee83384729a778675c1c0302f2, State: Running, Role: LEADER
I20260924 10:43:00.268767 18477 consensus_queue.cc:237] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:00.272598 18002 catalog_manager.cc:5495] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91ba2dee83384729a778675c1c0302f2 (127.2.8.3). New cstate: current_term: 1 leader_uuid: "91ba2dee83384729a778675c1c0302f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } health_report { overall_health: UNKNOWN } } }
W20260924 10:43:00.276203 18329 tablet.cc:2369] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:43:00.327479 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:00.324113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50868 (local address 127.2.8.3:36885)
0924 10:43:00.324236 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:00.324240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:00.324252 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:43:00.324381 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:00.324385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:00.324452 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:00.324546 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:43:00.324764 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.325350 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.326167 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:00.326380 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:00.326467 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:00.326722 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:43:00.327152 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:43:00.327243 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:00.327295 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 10:43:00.327346 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:43:00.333745 18144 raft_consensus.cc:1261] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 1 FOLLOWER]: Refusing update from remote peer 91ba2dee83384729a778675c1c0302f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:43:00.333822 18282 raft_consensus.cc:1261] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 1 FOLLOWER]: Refusing update from remote peer 91ba2dee83384729a778675c1c0302f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:43:00.334491 18477 consensus_queue.cc:1035] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:43:00.334692 18481 consensus_queue.cc:1035] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:43:00.342588 18484 mvcc.cc:205] Tried to move back new op lower bound from 7332849993039187968 to 7332849992783691776. Current Snapshot: MvccSnapshot[applied={T|T < 7332849993039187968}]
I20260924 10:43:00.342795 18485 mvcc.cc:205] Tried to move back new op lower bound from 7332849993039187968 to 7332849992783691776. Current Snapshot: MvccSnapshot[applied={T|T < 7332849993039187968}]
I20260924 10:43:00.350595 18002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:43:00.350775 18002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:43:00.353263 18002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Sending DeleteTablet for 3 replicas of tablet 54c76aae1f6747298a3a64c7b8c462b8
I20260924 10:43:00.353914 18262 tablet_service.cc:1514] Processing DeleteTablet for tablet 54c76aae1f6747298a3a64c7b8c462b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:00 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:57090
I20260924 10:43:00.354156 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 18040
I20260924 10:43:00.354187 18395 tablet_service.cc:1514] Processing DeleteTablet for tablet 54c76aae1f6747298a3a64c7b8c462b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:00 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:50864
I20260924 10:43:00.361192 18493 tablet_replica.cc:330] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7: stopping tablet replica
I20260924 10:43:00.361577 18493 raft_consensus.cc:2229] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 10:43:00.361619 17988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.8.1:34223 (error 108)
I20260924 10:43:00.361809 18494 tablet_replica.cc:330] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: stopping tablet replica
I20260924 10:43:00.361939 18493 raft_consensus.cc:2258] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:43:00.362393 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 18194
I20260924 10:43:00.363143 18494 raft_consensus.cc:2229] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:43:00.363595 18494 raft_consensus.cc:2258] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:43:00.365116 18494 ts_tablet_manager.cc:1905] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:43:00.367986 18494 ts_tablet_manager.cc:1918] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:43:00.368114 18494 log.cc:1196] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/54c76aae1f6747298a3a64c7b8c462b8
I20260924 10:43:00.368460 18494 ts_tablet_manager.cc:1939] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: Deleting consensus metadata
I20260924 10:43:00.369063 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 18331
I20260924 10:43:00.371235 18197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:00.370946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:44075 (local address 127.0.0.1:49810)
0924 10:43:00.371094 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:00.371157 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.2:44075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 10:43:00.371279 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:00.370886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:34223 (local address 127.0.0.1:55908)
0924 10:43:00.371109 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:00.371185 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:34223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260924 10:43:00.371393 17986 catalog_manager.cc:4584] TS 1a23cf3ba12b40eb949ba189063e3fd7 (127.2.8.2:44075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 54c76aae1f6747298a3a64c7b8c462b8: Network error: Client connection negotiation failed: client connection to 127.2.8.2:44075: connect: Connection refused (error 111)
I20260924 10:43:00.375059 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 17969
I20260924 10:43:20.382431 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:45591
--webserver_interface=127.2.8.62
--webserver_port=46579
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:45591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:20.497488 18496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:20.497743 18496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:20.497783 18496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:20.501890 18496 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 10:43:20.501948 18496 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 10:43:20.501963 18496 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 10:43:20.501977 18496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:43:20.501995 18496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:20.502012 18496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:20.502027 18496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:43:20.502043 18496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:43:20.507362 18496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:45591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:45591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=46579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:20.508427 18496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:20.509402 18496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:20.518028 18504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:20.518069 18502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:20.518069 18501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:20.518153 18496 server_base.cc:1034] running on GCE node
I20260924 10:43:20.518806 18496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:20.519856 18496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:20.521063 18496 hybrid_clock.cc:648] HybridClock initialized: now 1790246600521034 us; error 51 us; skew 500 ppm
Sep 24 10:43:20 dist-test-slave-j38h krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:20.528614 18496 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:43:20.529644 18496 webserver.cc:460] Webserver started at http://127.2.8.62:46579/ using document root <none> and password file <none>
I20260924 10:43:20.530239 18496 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:20.530336 18496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:20.534322 18496 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:43:20.536451 18511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:20.537822 18496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:43:20.537992 18496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "47797fad5d7e4511b1040bcbaa4b7048"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "5bac52b6c910de4bc9ca86c2bd38229d"
server_key_iv: "a0e8cf6d8d60a5ada6de1b2fb3c658e5"
server_key_version: "encryptionkey@0"
I20260924 10:43:20.538582 18496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:20.559525 18496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:20.563373 18496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:20.563607 18496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:20.571556 18496 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:45591
I20260924 10:43:20.571867 18563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:45591 every 8 connection(s)
I20260924 10:43:20.573096 18496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:43:20.577502 18564 sys_catalog.cc:263] Verifying existing consensus state
I20260924 10:43:20.579133 18564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Bootstrap starting.
I20260924 10:43:20.579645 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 18496
I20260924 10:43:20.580156 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:34223
--local_ip_for_outbound_sockets=127.2.8.1
--tserver_master_addrs=127.2.8.62:45591
--webserver_port=44557
--webserver_interface=127.2.8.1
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:43:20.597239 18564 log.cc:824] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:20.608301 18564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:43:20.608711 18564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Bootstrap complete.
I20260924 10:43:20.612721 18564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } }
I20260924 10:43:20.613191 18564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47797fad5d7e4511b1040bcbaa4b7048, State: Initialized, Role: FOLLOWER
I20260924 10:43:20.613665 18564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } }
I20260924 10:43:20.613824 18564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:43:20.613914 18564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:43:20.614037 18564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 2 FOLLOWER]: Advancing to term 3
I20260924 10:43:20.615865 18564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } }
I20260924 10:43:20.616211 18564 leader_election.cc:302] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47797fad5d7e4511b1040bcbaa4b7048; no voters:
I20260924 10:43:20.616539 18564 leader_election.cc:288] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 10:43:20.616617 18568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 10:43:20.616797 18568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [term 3 LEADER]: Becoming Leader. State: Replica: 47797fad5d7e4511b1040bcbaa4b7048, State: Running, Role: LEADER
I20260924 10:43:20.617107 18568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } }
I20260924 10:43:20.617769 18564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:43:20.618019 18568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47797fad5d7e4511b1040bcbaa4b7048. Latest consensus state: current_term: 3 leader_uuid: "47797fad5d7e4511b1040bcbaa4b7048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } } }
I20260924 10:43:20.618135 18568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:20.618554 18569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "47797fad5d7e4511b1040bcbaa4b7048" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47797fad5d7e4511b1040bcbaa4b7048" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45591 } } }
I20260924 10:43:20.618728 18569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048 [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:20.619812 18572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:43:20.625335 18572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:43:20.625989 18572 catalog_manager.cc:1259] Loaded cluster ID: fc2ea8eb88ae49978761aca9042f1b9e
I20260924 10:43:20.626052 18572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:43:20.628671 18572 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:43:20.629518 18572 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Loaded TSK: 0
I20260924 10:43:20.629624 18572 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 47797fad5d7e4511b1040bcbaa4b7048: Loaded TSK: 1
F20260924 10:43:20.630617 18572 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbcd60599ad google::LogMessage::Fail() at ??:0
@ 0x7fbcd605c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fbcd60594de google::LogMessage::Flush() at ??:0
@ 0x7fbcd605aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbcd8d1d18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fbcd8d15f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fbce09be7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fbce099899c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fbce0999178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fbce09dadf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fbce09f982b std::function<>::operator()() at ??:0
@ 0x7fbce0998b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fbce0999f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbce0996579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fbce09da640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbcd82ab3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbcd82abc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fbcd82ad3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbcd829c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbcd98e06db start_thread at ??:0
@ 0x7fbcd4b5761f clone at ??:0
W20260924 10:43:20.745261 18566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:20.745537 18566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:20.745585 18566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:20.749683 18566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:20.749740 18566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:20.749832 18566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:43:20.755182 18566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:34223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=44557
--tserver_master_addrs=127.2.8.62:45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:20.756284 18566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:20.757373 18566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:20.768283 18611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:20.768265 18614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:20.768258 18612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:20.768567 18566 server_base.cc:1034] running on GCE node
I20260924 10:43:20.769147 18566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:20.769831 18566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:20.771035 18566 hybrid_clock.cc:648] HybridClock initialized: now 1790246600771007 us; error 47 us; skew 500 ppm
Sep 24 10:43:20 dist-test-slave-j38h krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:20.773988 18566 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:43:20.774923 18566 webserver.cc:460] Webserver started at http://127.2.8.1:44557/ using document root <none> and password file <none>
I20260924 10:43:20.775496 18566 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:20.775590 18566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:20.779667 18566 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 10:43:20.782119 18621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:20.783435 18566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:43:20.783612 18566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "e28a7ed37a6f41738f3d48228a072c2d"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "5741d069547885c52b100e3e17e97476"
server_key_iv: "cb9d77d7189dcceff5e9131d9963baf0"
server_key_version: "encryptionkey@0"
I20260924 10:43:20.784176 18566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:20.793538 18566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:20.796602 18566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:20.796799 18566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:20.797426 18566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:20.798519 18628 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 10:43:20.803884 18566 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:43:20.803992 18566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 10:43:20.804072 18566 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:43:20.805805 18566 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:43:20.805876 18566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260924 10:43:20.806181 18628 tablet_bootstrap.cc:492] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d: Bootstrap starting.
I20260924 10:43:20.816437 18566 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:34223
I20260924 10:43:20.816504 18735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:34223 every 8 connection(s)
I20260924 10:43:20.817430 18566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:43:20.818017 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 18566
I20260924 10:43:20.818539 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:44075
--local_ip_for_outbound_sockets=127.2.8.2
--tserver_master_addrs=127.2.8.62:45591
--webserver_port=39115
--webserver_interface=127.2.8.2
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:43:20.829105 18628 log.cc:824] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:20.835974 18628 tablet_bootstrap.cc:492] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:43:20.836380 18628 tablet_bootstrap.cc:492] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d: Bootstrap complete.
I20260924 10:43:20.837574 18628 ts_tablet_manager.cc:1397] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.008s
I20260924 10:43:20.840065 18628 raft_consensus.cc:348] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:20.840531 18628 raft_consensus.cc:729] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28a7ed37a6f41738f3d48228a072c2d, State: Initialized, Role: FOLLOWER
I20260924 10:43:20.841068 18628 consensus_queue.cc:260] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:20.842015 18628 ts_tablet_manager.cc:1428] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260924 10:43:20.869177 18739 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:43:20.820199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:34557)
0924 10:43:20.820669 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:20.820723 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:20.821819 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:20.868206 (+ 46387us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":129,"threads_started":1}
W20260924 10:43:20.869726 18736 heartbeater.cc:644] Failed to heartbeat to 127.2.8.62:45591 (0 consecutive failures): Network error: Failed to ping master at 127.2.8.62:45591: Client connection negotiation failed: client connection to 127.2.8.62:45591: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 10:43:20.870553 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:20.870286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:42311)
0924 10:43:20.870427 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:20.870495 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 10:43:20.871273 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:20.871037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:33249)
0924 10:43:20.871159 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:20.871212 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 10:43:20.871498 18736 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 10:43:20.951174 18738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:20.951489 18738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:20.951536 18738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:20.955709 18738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:20.955783 18738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:20.955845 18738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:43:20.961266 18738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:44075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=39115
--tserver_master_addrs=127.2.8.62:45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:20.962386 18738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:20.963454 18738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:20.973555 18747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:20.973423 18738 server_base.cc:1034] running on GCE node
W20260924 10:43:20.973453 18746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:20.973431 18749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:20.974207 18738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:20.974850 18738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:20.976027 18738 hybrid_clock.cc:648] HybridClock initialized: now 1790246600975990 us; error 48 us; skew 500 ppm
Sep 24 10:43:20 dist-test-slave-j38h krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:20.978736 18738 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:43:20.979619 18738 webserver.cc:460] Webserver started at http://127.2.8.2:39115/ using document root <none> and password file <none>
I20260924 10:43:20.980144 18738 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:20.980203 18738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:20.983978 18738 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:43:20.986152 18756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:20.987711 18738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 10:43:20.987807 18738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "1a23cf3ba12b40eb949ba189063e3fd7"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "3ad746a293de0c9bb70a4206bd1604f8"
server_key_iv: "ccac7ddd65b64fcc2337a3e05d95e91d"
server_key_version: "encryptionkey@0"
I20260924 10:43:20.988412 18738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:21.032326 18738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:21.035851 18738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:21.036058 18738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:21.036706 18738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:21.037889 18763 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 10:43:21.042304 18738 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 10:43:21.042371 18738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260924 10:43:21.042464 18738 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 10:43:21.044139 18738 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 10:43:21.044183 18738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 10:43:21.044508 18763 tablet_bootstrap.cc:492] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7: Bootstrap starting.
I20260924 10:43:21.054833 18738 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:44075
I20260924 10:43:21.054909 18870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:44075 every 8 connection(s)
I20260924 10:43:21.055828 18738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:43:21.059562 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:21.057846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:44557)
0924 10:43:21.058428 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:21.058595 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":128,"threads_started":1}
W20260924 10:43:21.060081 18871 heartbeater.cc:644] Failed to heartbeat to 127.2.8.62:45591 (0 consecutive failures): Network error: Failed to ping master at 127.2.8.62:45591: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
I20260924 10:43:21.060712 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 18738
I20260924 10:43:21.060798 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:21.060547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:41335)
0924 10:43:21.060678 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:21.060735 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:43:21.061244 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:36885
--local_ip_for_outbound_sockets=127.2.8.3
--tserver_master_addrs=127.2.8.62:45591
--webserver_port=45119
--webserver_interface=127.2.8.3
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:43:21.063172 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:21.062896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:37603)
0924 10:43:21.063034 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:21.063095 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 10:43:21.063360 18871 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:43:21.063584 18763 log.cc:824] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:21.070948 18763 tablet_bootstrap.cc:492] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:43:21.071359 18763 tablet_bootstrap.cc:492] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7: Bootstrap complete.
I20260924 10:43:21.072685 18763 ts_tablet_manager.cc:1397] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.008s
I20260924 10:43:21.075549 18763 raft_consensus.cc:348] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:21.076000 18763 raft_consensus.cc:729] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a23cf3ba12b40eb949ba189063e3fd7, State: Initialized, Role: FOLLOWER
I20260924 10:43:21.076476 18763 consensus_queue.cc:260] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:21.077567 18763 ts_tablet_manager.cc:1428] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260924 10:43:21.077826 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:21.077515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:53991)
0924 10:43:21.077666 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:21.077742 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 10:43:21.078663 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:21.078429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:51503)
0924 10:43:21.078546 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:21.078601 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:43:21.173563 18874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:21.173825 18874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:21.173869 18874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:21.177990 18874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:21.178045 18874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:21.178103 18874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:43:21.183509 18874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:36885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=45119
--tserver_master_addrs=127.2.8.62:45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:21.184501 18874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:21.185541 18874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:43:21.195768 18874 server_base.cc:1034] running on GCE node
W20260924 10:43:21.195796 18884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:21.195777 18882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:21.195777 18881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:21.196568 18874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:21.197436 18874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:21.198649 18874 hybrid_clock.cc:648] HybridClock initialized: now 1790246601198611 us; error 48 us; skew 500 ppm
Sep 24 10:43:21 dist-test-slave-j38h krb5kdc[17420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246601, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:21.201890 18874 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:43:21.202852 18874 webserver.cc:460] Webserver started at http://127.2.8.3:45119/ using document root <none> and password file <none>
I20260924 10:43:21.203423 18874 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:21.203485 18874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:21.207665 18874 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:43:21.209890 18891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:21.211239 18874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:43:21.211341 18874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "91ba2dee83384729a778675c1c0302f2"
format_stamp: "Formatted at 2026-09-24 10:42:38 on dist-test-slave-j38h"
server_key: "0dc19bd34d34f25f3b7d2be9c25bd5d9"
server_key_iv: "225f9ea178b2cb44db1c9adca1303438"
server_key_version: "encryptionkey@0"
I20260924 10:43:21.211877 18874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:21.226794 18874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:21.230181 18874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:21.230338 18874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:21.230878 18874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:21.231657 18874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:21.231702 18874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:21.231740 18874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:21.231756 18874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:21.241225 18874 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:36885
I20260924 10:43:21.241245 19004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:36885 every 8 connection(s)
I20260924 10:43:21.242312 18874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:43:21.246047 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:21.244506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:55965)
0924 10:43:21.245221 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:21.245329 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":108,"threads_started":1}
W20260924 10:43:21.246568 19005 heartbeater.cc:644] Failed to heartbeat to 127.2.8.62:45591 (0 consecutive failures): Network error: Failed to ping master at 127.2.8.62:45591: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
I20260924 10:43:21.247169 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:21.246948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:53413)
0924 10:43:21.247071 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:21.247116 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 10:43:21.247780 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:21.247605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:35127)
0924 10:43:21.247697 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:21.247737 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 10:43:21.247967 19005 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 10:43:21.248208 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 18874
I20260924 10:43:21.872737 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:21.872252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:37213)
0924 10:43:21.872552 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:21.872618 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":83,"threads_started":1}
I20260924 10:43:22.079891 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:22.079386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:42983)
0924 10:43:22.079686 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:22.079765 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":124,"threads_started":1}
I20260924 10:43:22.189019 19011 raft_consensus.cc:482] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:43:22.189247 19011 raft_consensus.cc:504] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:22.190411 19011 leader_election.cc:288] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1a23cf3ba12b40eb949ba189063e3fd7 (127.2.8.2:44075), 91ba2dee83384729a778675c1c0302f2 (127.2.8.3:36885)
Sep 24 10:43:22 dist-test-slave-j38h krb5kdc[17420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.3@KRBTEST.COM
Sep 24 10:43:22 dist-test-slave-j38h krb5kdc[17420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.2@KRBTEST.COM
I20260924 10:43:22.204238 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:22.190783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:55357 (local address 127.2.8.2:44075)
0924 10:43:22.191193 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:22.191206 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:22.191243 (+ 37us) server_negotiation.cc:407] Connection header received
0924 10:43:22.191502 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:22.191506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:22.191850 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:22.192150 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:22.193580 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:22.194525 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:22.195764 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:22.196200 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:22.196384 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:22.199486 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:22.199491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:22.199502 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:22.199527 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:22.201310 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:22.201890 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:22.201895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:22.201899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:22.202017 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:22.202369 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:22.202375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:22.202378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:22.202902 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:22.203060 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:22.203769 (+ 709us) server_negotiation.cc:299] Negotiation successful
0924 10:43:22.204055 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":64,"threads_started":1}
I20260924 10:43:22.204249 19014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:22.190804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:33765 (local address 127.2.8.3:36885)
0924 10:43:22.191223 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:22.191239 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:22.191280 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:43:22.192030 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:22.192039 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:22.192425 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:22.192712 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:22.193561 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:22.194515 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:22.195728 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:22.196183 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:22.196336 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:22.199102 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:22.199107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:22.199117 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:22.199143 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:22.201200 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:22.202149 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:22.202154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:22.202157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:22.202256 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:22.202547 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:22.202550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:22.202552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:22.202971 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:22.203080 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:22.203769 (+ 689us) server_negotiation.cc:299] Negotiation successful
0924 10:43:22.204055 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":110,"threads_started":1}
I20260924 10:43:22.204305 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:22.190597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:44075 (local address 127.2.8.1:55357)
0924 10:43:22.190804 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:22.190819 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:22.190936 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:22.192197 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:22.192203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:22.192877 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:22.193398 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:22.193415 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:22.194688 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:22.194690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:22.195512 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:22.195520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:22.196340 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:22.196356 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:22.196537 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:22.197208 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:43:22.197223 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:22.199274 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:22.201420 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:22.201425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:22.201429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:22.201761 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:22.202119 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:22.202121 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:22.202123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:22.202249 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:22.203059 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:22.203065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:22.203399 (+ 334us) client_negotiation.cc:769] Sending connection context
0924 10:43:22.203631 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 10:43:22.204093 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":327,"spinlock_wait_cycles":21888}
I20260924 10:43:22.204304 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:22.190633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:36885 (local address 127.2.8.1:33765)
0924 10:43:22.190921 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:22.190932 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:22.191019 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:22.192682 (+ 1663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:22.192684 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:22.192920 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:22.193398 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:22.193413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:22.194654 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:22.194660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:22.195514 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:22.195523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:22.196292 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:22.196313 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:22.196535 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:22.197265 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:43:22.197277 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:22.198887 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:22.201329 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:22.201335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:22.201347 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:22.201761 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:22.202368 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:22.202371 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:22.202373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:22.202450 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:22.203062 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:22.203067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:22.203399 (+ 332us) client_negotiation.cc:769] Sending connection context
0924 10:43:22.203631 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 10:43:22.204040 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":89,"spinlock_wait_cycles":28672,"thread_start_us":69,"threads_started":1}
I20260924 10:43:22.205768 18824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c76aae1f6747298a3a64c7b8c462b8" candidate_uuid: "e28a7ed37a6f41738f3d48228a072c2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" is_pre_election: true
I20260924 10:43:22.206105 18824 raft_consensus.cc:2454] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e28a7ed37a6f41738f3d48228a072c2d in term 1.
I20260924 10:43:22.207278 18622 leader_election.cc:302] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a23cf3ba12b40eb949ba189063e3fd7, e28a7ed37a6f41738f3d48228a072c2d; no voters:
W20260924 10:43:22.207535 18624 leader_election.cc:341] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91ba2dee83384729a778675c1c0302f2 (127.2.8.3:36885): Not found: Tablet not found: 54c76aae1f6747298a3a64c7b8c462b8
I20260924 10:43:22.207630 19011 raft_consensus.cc:2781] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 10:43:22.207759 19011 raft_consensus.cc:482] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:43:22.207830 19011 raft_consensus.cc:3037] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 1 FOLLOWER]: Advancing to term 2
I20260924 10:43:22.210240 19011 raft_consensus.cc:504] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:22.210613 19011 leader_election.cc:288] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [CANDIDATE]: Term 2 election: Requested vote from peers 1a23cf3ba12b40eb949ba189063e3fd7 (127.2.8.2:44075), 91ba2dee83384729a778675c1c0302f2 (127.2.8.3:36885)
I20260924 10:43:22.211014 18824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c76aae1f6747298a3a64c7b8c462b8" candidate_uuid: "e28a7ed37a6f41738f3d48228a072c2d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1a23cf3ba12b40eb949ba189063e3fd7"
I20260924 10:43:22.211158 18824 raft_consensus.cc:3037] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 1 FOLLOWER]: Advancing to term 2
W20260924 10:43:22.211277 18624 leader_election.cc:341] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 91ba2dee83384729a778675c1c0302f2 (127.2.8.3:36885): Not found: Tablet not found: 54c76aae1f6747298a3a64c7b8c462b8
I20260924 10:43:22.213255 18824 raft_consensus.cc:2454] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e28a7ed37a6f41738f3d48228a072c2d in term 2.
I20260924 10:43:22.213677 18622 leader_election.cc:302] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1a23cf3ba12b40eb949ba189063e3fd7, e28a7ed37a6f41738f3d48228a072c2d; no voters: 91ba2dee83384729a778675c1c0302f2
I20260924 10:43:22.213892 19011 raft_consensus.cc:2781] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 2 FOLLOWER]: Leader election won for term 2
I20260924 10:43:22.214099 19011 raft_consensus.cc:686] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [term 2 LEADER]: Becoming Leader. State: Replica: e28a7ed37a6f41738f3d48228a072c2d, State: Running, Role: LEADER
I20260924 10:43:22.214448 19011 consensus_queue.cc:237] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:22.214883 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:22.214576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:35949)
0924 10:43:22.214752 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:22.214808 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 10:43:22.249051 19017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:22.248544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:57073)
0924 10:43:22.248863 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:22.248921 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":68,"threads_started":1}
I20260924 10:43:22.745462 18824 raft_consensus.cc:1261] T 54c76aae1f6747298a3a64c7b8c462b8 P 1a23cf3ba12b40eb949ba189063e3fd7 [term 2 FOLLOWER]: Refusing update from remote peer e28a7ed37a6f41738f3d48228a072c2d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260924 10:43:22.746394 19018 consensus_queue.cc:1035] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 10:43:22.748596 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:22.748216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:35775)
0924 10:43:22.748427 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:22.748493 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":55,"threads_started":1}
W20260924 10:43:22.751451 18624 consensus_peers.cc:487] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d -> Peer 91ba2dee83384729a778675c1c0302f2 (127.2.8.3:36885): Couldn't send request to peer 91ba2dee83384729a778675c1c0302f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54c76aae1f6747298a3a64c7b8c462b8. This is attempt 1: this message will repeat every 5th retry.
I20260924 10:43:22.751497 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:22.750931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:48271)
0924 10:43:22.751334 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:22.751398 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":50,"threads_started":1}
I20260924 10:43:22.751582 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:22.751316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:39567)
0924 10:43:22.751454 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:22.751510 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 10:43:23.154663 19026 ts_tablet_manager.cc:927] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: Initiating tablet copy from peer e28a7ed37a6f41738f3d48228a072c2d (127.2.8.1:34223)
I20260924 10:43:23.155320 19026 tablet_copy_client.cc:316] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: tablet copy: Beginning tablet copy session from remote peer at address 127.2.8.1:34223
Sep 24 10:43:23 dist-test-slave-j38h krb5kdc[17420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246601, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.1@KRBTEST.COM
I20260924 10:43:23.165807 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:23.155870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:34223 (local address 127.2.8.3:53965)
0924 10:43:23.156133 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:23.156162 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:23.156288 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:23.156872 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:23.156877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:23.157280 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:23.157419 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:23.157426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:23.158507 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:23.158510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:23.159185 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:23.159207 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:23.159851 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:23.159867 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:23.159963 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:23.160643 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:43:23.160655 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:23.162367 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:23.164140 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:23.164147 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:23.164152 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:23.164364 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:23.164675 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:23.164678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:23.164679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:23.164752 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:23.165325 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:23.165330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:23.165435 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 10:43:23.165555 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 10:43:23.165679 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":119,"threads_started":1}
I20260924 10:43:23.166041 19029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:23.156041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:53965 (local address 127.2.8.1:34223)
0924 10:43:23.156438 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:23.156449 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:23.156472 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:43:23.156521 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:23.156532 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:23.156713 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:23.156842 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:23.157548 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:23.158376 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:23.159350 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:23.159732 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:23.159830 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:23.162508 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:23.162517 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:23.162527 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:23.162551 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:23.164041 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:23.164493 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:23.164498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:23.164500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:23.164586 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:23.164875 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:23.164878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:23.164880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:23.165218 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:23.165375 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:23.165674 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 10:43:23.165859 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":84,"threads_started":1}
I20260924 10:43:23.167070 18710 tablet_copy_service.cc:140] P e28a7ed37a6f41738f3d48228a072c2d: Received BeginTabletCopySession request for tablet 54c76aae1f6747298a3a64c7b8c462b8 from peer 91ba2dee83384729a778675c1c0302f2 ({username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:53965)
I20260924 10:43:23.167238 18710 tablet_copy_service.cc:161] P e28a7ed37a6f41738f3d48228a072c2d: Beginning new tablet copy session on tablet 54c76aae1f6747298a3a64c7b8c462b8 from peer 91ba2dee83384729a778675c1c0302f2 at {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:53965: session id = 91ba2dee83384729a778675c1c0302f2-54c76aae1f6747298a3a64c7b8c462b8
I20260924 10:43:23.168440 18710 tablet_copy_source_session.cc:205] T 54c76aae1f6747298a3a64c7b8c462b8 P e28a7ed37a6f41738f3d48228a072c2d: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 10:43:23.172351 19026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c76aae1f6747298a3a64c7b8c462b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:23.178731 19026 tablet_copy_client.cc:791] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: tablet copy: Starting download of 0 data blocks...
I20260924 10:43:23.178958 19026 tablet_copy_client.cc:655] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: tablet copy: Starting download of 1 WAL segments...
I20260924 10:43:23.180691 19026 tablet_copy_client.cc:531] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 10:43:23.182335 19026 tablet_bootstrap.cc:492] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: Bootstrap starting.
I20260924 10:43:23.187810 19026 log.cc:824] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:23.194335 19026 tablet_bootstrap.cc:492] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 10:43:23.194674 19026 tablet_bootstrap.cc:492] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: Bootstrap complete.
I20260924 10:43:23.194955 19026 ts_tablet_manager.cc:1397] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260924 10:43:23.197190 19026 raft_consensus.cc:348] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:23.197420 19026 raft_consensus.cc:729] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91ba2dee83384729a778675c1c0302f2, State: Initialized, Role: FOLLOWER
I20260924 10:43:23.197783 19026 consensus_queue.cc:260] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a23cf3ba12b40eb949ba189063e3fd7" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44075 } } peers { permanent_uuid: "91ba2dee83384729a778675c1c0302f2" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 36885 } } peers { permanent_uuid: "e28a7ed37a6f41738f3d48228a072c2d" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 34223 } }
I20260924 10:43:23.198824 19026 ts_tablet_manager.cc:1428] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260924 10:43:23.198967 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:23.198713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:51509)
0924 10:43:23.198822 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:23.198882 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 10:43:23.199561 18710 tablet_copy_service.cc:342] P e28a7ed37a6f41738f3d48228a072c2d: Request end of tablet copy session 91ba2dee83384729a778675c1c0302f2-54c76aae1f6747298a3a64c7b8c462b8 received from {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:53965
I20260924 10:43:23.199688 18710 tablet_copy_service.cc:433] P e28a7ed37a6f41738f3d48228a072c2d: ending tablet copy session 91ba2dee83384729a778675c1c0302f2-54c76aae1f6747298a3a64c7b8c462b8 on tablet 54c76aae1f6747298a3a64c7b8c462b8 with peer 91ba2dee83384729a778675c1c0302f2
I20260924 10:43:23.199887 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:23.199649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:41803)
0924 10:43:23.199792 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:23.199838 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 10:43:23.596107 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:23.595752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:56677)
0924 10:43:23.595957 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:23.596014 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260924 10:43:23.596091 18959 raft_consensus.cc:1203] T 54c76aae1f6747298a3a64c7b8c462b8 P 91ba2dee83384729a778675c1c0302f2 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260924 10:43:23.597159 19028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:23.596855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:60287)
0924 10:43:23.597015 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:23.597073 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:43:23.752599 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:23.752229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:47899)
0924 10:43:23.752451 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:23.752506 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
I20260924 10:43:24.597285 19036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:24.596845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:60775)
0924 10:43:24.597131 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:24.597189 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":67,"threads_started":1}
I20260924 10:43:24.598121 19037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:24.597792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:43179)
0924 10:43:24.598000 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:24.598047 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":60,"threads_started":1}
I20260924 10:43:24.754060 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:24.753554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:59005)
0924 10:43:24.753877 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:24.753951 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":71,"threads_started":1}
I20260924 10:43:25.598546 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:25.598120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:51259)
0924 10:43:25.598382 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:25.598443 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260924 10:43:25.599316 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:25.598877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:52289)
0924 10:43:25.599177 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:25.599239 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":90,"threads_started":1}
I20260924 10:43:25.755379 19041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:25.754942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:35095)
0924 10:43:25.755216 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:25.755279 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
I20260924 10:43:26.600201 19042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:26.599696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:36469)
0924 10:43:26.600053 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:26.600105 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":70,"threads_started":1}
I20260924 10:43:26.600589 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:26.600192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:36159)
0924 10:43:26.600403 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:26.600464 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
I20260924 10:43:26.600909 19042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:26.600682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:42515)
0924 10:43:26.600797 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:26.600847 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 10:43:26.756767 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:26.756339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:42669)
0924 10:43:26.756616 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:26.756672 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20260924 10:43:27.601981 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:27.601518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:60269)
0924 10:43:27.601796 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:27.601879 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
I20260924 10:43:27.602012 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:27.601594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:37321)
0924 10:43:27.601866 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:27.601916 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":84,"threads_started":1}
I20260924 10:43:27.758134 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:27.757633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:45005)
0924 10:43:27.757959 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:27.758028 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":86,"threads_started":1}
I20260924 10:43:28.603205 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:28.602792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:55359)
0924 10:43:28.603069 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:28.603121 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":100,"threads_started":1}
I20260924 10:43:28.603206 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:28.602785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:56835)
0924 10:43:28.603056 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:28.603112 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":66,"threads_started":1}
I20260924 10:43:28.759377 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:28.758953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:37731)
0924 10:43:28.759221 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:28.759279 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":73,"threads_started":1}
I20260924 10:43:29.604394 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:29.603991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:33355)
0924 10:43:29.604248 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:29.604299 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":106,"threads_started":1}
I20260924 10:43:29.604660 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:29.604144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:33699)
0924 10:43:29.604441 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:29.604537 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":102,"threads_started":1}
I20260924 10:43:29.760692 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:29.760227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:50573)
0924 10:43:29.760526 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:29.760586 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":131,"threads_started":1}
I20260924 10:43:30.605818 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:30.605235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:58359)
0924 10:43:30.605644 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:30.605712 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":96,"threads_started":1}
I20260924 10:43:30.605863 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:30.605457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:55869)
0924 10:43:30.605694 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:30.605755 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":91,"threads_started":1}
I20260924 10:43:30.762038 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:30.761605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:51563)
0924 10:43:30.761882 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:30.761940 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":71,"threads_started":1}
I20260924 10:43:31.607177 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:31.606740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:47427)
0924 10:43:31.606992 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:31.607061 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":93,"threads_started":1}
I20260924 10:43:31.607236 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:31.606844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:60495)
0924 10:43:31.607099 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:31.607149 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260924 10:43:31.763559 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:31.763053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:50251)
0924 10:43:31.763349 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:31.763412 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":121,"threads_started":1}
I20260924 10:43:32.608593 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:32.608057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:34187)
0924 10:43:32.608366 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:32.608441 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":105,"threads_started":1}
I20260924 10:43:32.608593 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:32.608060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:41457)
0924 10:43:32.608434 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:32.608488 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":136,"threads_started":1}
I20260924 10:43:32.764994 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:32.764471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:48975)
0924 10:43:32.764777 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:32.764843 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":114,"threads_started":1}
I20260924 10:43:33.610018 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:33.609559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:48559)
0924 10:43:33.609847 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:33.609910 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":87,"threads_started":1}
I20260924 10:43:33.610044 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:33.609559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:37129)
0924 10:43:33.609903 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:33.609957 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":88,"threads_started":1}
I20260924 10:43:33.766508 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:33.765919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:38437)
0924 10:43:33.766283 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:33.766353 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":122,"threads_started":1}
I20260924 10:43:34.611327 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:34.610897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:53559)
0924 10:43:34.611171 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:34.611227 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":86,"threads_started":1}
I20260924 10:43:34.611357 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:34.610892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:55121)
0924 10:43:34.611223 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:34.611274 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":89,"threads_started":1}
I20260924 10:43:34.768138 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:34.767401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:55831)
0924 10:43:34.767866 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:34.767979 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":101,"threads_started":1}
I20260924 10:43:35.612671 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:35.612165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.1:36037)
0924 10:43:35.612508 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:35.612571 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":88,"threads_started":1}
I20260924 10:43:35.612725 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:35.612203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.3:38939)
0924 10:43:35.612563 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:35.612639 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":110,"threads_started":1}
I20260924 10:43:35.769546 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:35.769039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:45591 (local address 127.2.8.2:55865)
0924 10:43:35.769373 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:35.769440 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.62:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":69,"threads_started":1}
I20260924 10:43:36.249724 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 18566
I20260924 10:43:36.255538 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 18738
I20260924 10:43:36.260826 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 18874
2026-09-24T10:43:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59469 ms)
[----------] 30 tests from SecurityITest (241435 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:43:36 dist-test-slave-j38h krb5kdc[19075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:43:36 dist-test-slave-j38h krb5kdc[19075](info): set up 2 sockets
Sep 24 10:43:36 dist-test-slave-j38h krb5kdc[19075](info): commencing operation
krb5kdc: starting...
W20260924 10:43:37.316802 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:43:37 dist-test-slave-j38h krb5kdc[19075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:43:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:43:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:43:37.445304 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:42041
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:42041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:37.555114 19088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:37.555369 19088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:37.555409 19088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:37.559458 19088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:43:37.559513 19088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:37.559532 19088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:37.559549 19088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:43:37.559563 19088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:43:37.564992 19088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:42041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:42041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:37.566133 19088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:37.567216 19088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:37.575968 19094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:37.576053 19088 server_base.cc:1034] running on GCE node
W20260924 10:43:37.575977 19096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:37.575990 19093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:37.576795 19088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:37.577963 19088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:37.579169 19088 hybrid_clock.cc:648] HybridClock initialized: now 1790246617579150 us; error 30 us; skew 500 ppm
Sep 24 10:43:37 dist-test-slave-j38h krb5kdc[19075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246617, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:37.582118 19088 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:43:37.583014 19088 webserver.cc:460] Webserver started at http://127.2.8.62:36821/ using document root <none> and password file <none>
I20260924 10:43:37.583632 19088 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:37.583727 19088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:37.583971 19088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:37.586110 19088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "71fa946349ad47e9ac86f74946f42977"
format_stamp: "Formatted at 2026-09-24 10:43:37 on dist-test-slave-j38h"
server_key: "3077133ede5caa832d597726052c124c"
server_key_iv: "f5a7b3522db1497fc502faf21d6f0aff"
server_key_version: "encryptionkey@0"
I20260924 10:43:37.586680 19088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "71fa946349ad47e9ac86f74946f42977"
format_stamp: "Formatted at 2026-09-24 10:43:37 on dist-test-slave-j38h"
server_key: "3077133ede5caa832d597726052c124c"
server_key_iv: "f5a7b3522db1497fc502faf21d6f0aff"
server_key_version: "encryptionkey@0"
I20260924 10:43:37.590432 19088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:43:37.592762 19103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:37.594143 19088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:43:37.594444 19088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "71fa946349ad47e9ac86f74946f42977"
format_stamp: "Formatted at 2026-09-24 10:43:37 on dist-test-slave-j38h"
server_key: "3077133ede5caa832d597726052c124c"
server_key_iv: "f5a7b3522db1497fc502faf21d6f0aff"
server_key_version: "encryptionkey@0"
I20260924 10:43:37.594590 19088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:37.608126 19088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:37.611850 19088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:37.612087 19088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:37.619598 19088 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:42041
I20260924 10:43:37.619655 19155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:42041 every 8 connection(s)
I20260924 10:43:37.620782 19088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:43:37.621842 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 19088
I20260924 10:43:37.621989 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:43:37.622323 2080 external_mini_cluster.cc:1428] Setting key 1a5d3914f47680a907735d0c2f063866
I20260924 10:43:37.624325 19156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:37.629845 19156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977: Bootstrap starting.
Sep 24 10:43:37 dist-test-slave-j38h krb5kdc[19075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:43:37.632079 19156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:37.632936 19156 log.cc:824] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:37.635011 19156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977: No bootstrap required, opened a new log
I20260924 10:43:37.637501 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:37.623712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54954 (local address 127.2.8.62:42041)
0924 10:43:37.624158 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:37.624174 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:37.624210 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:43:37.625130 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:37.625150 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:37.625497 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:37.625854 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:37.626702 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:37.627762 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:37.628742 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:37.629309 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:37.629516 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:37.632171 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:37.632205 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:37.632227 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:37.632262 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:37.634410 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:37.634919 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:37.634924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:37.634928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:37.635044 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:37.635425 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:37.635432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:37.635434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:37.635836 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:37.636058 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:37.636374 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 10:43:37.636676 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":153,"threads_started":1}
I20260924 10:43:37.639000 19156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fa946349ad47e9ac86f74946f42977" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42041 } }
I20260924 10:43:37.639247 19156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:37.639343 19156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71fa946349ad47e9ac86f74946f42977, State: Initialized, Role: FOLLOWER
I20260924 10:43:37.639827 19156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fa946349ad47e9ac86f74946f42977" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42041 } }
I20260924 10:43:37.639986 19156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:43:37.640066 19156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:43:37.640138 19156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:37.641032 19156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fa946349ad47e9ac86f74946f42977" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42041 } }
I20260924 10:43:37.641430 19156 leader_election.cc:302] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71fa946349ad47e9ac86f74946f42977; no voters:
I20260924 10:43:37.641816 19156 leader_election.cc:288] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:43:37.641952 19161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:37.642204 19161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [term 1 LEADER]: Becoming Leader. State: Replica: 71fa946349ad47e9ac86f74946f42977, State: Running, Role: LEADER
I20260924 10:43:37.642571 19161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fa946349ad47e9ac86f74946f42977" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42041 } }
I20260924 10:43:37.643050 19156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:43:37.644331 19162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71fa946349ad47e9ac86f74946f42977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fa946349ad47e9ac86f74946f42977" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42041 } } }
I20260924 10:43:37.645087 19162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:37.644927 19163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71fa946349ad47e9ac86f74946f42977. Latest consensus state: current_term: 1 leader_uuid: "71fa946349ad47e9ac86f74946f42977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fa946349ad47e9ac86f74946f42977" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42041 } } }
I20260924 10:43:37.645723 19163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977 [sys.catalog]: This master's current role is: LEADER
W20260924 10:43:37.647913 19175 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:43:37.648016 19175 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:43:37.648094 19177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:43:37.648768 19177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:43:37.654049 19177 catalog_manager.cc:1347] Generated new cluster ID: b887dc609bf04fae9eed99aed24b1ad0
I20260924 10:43:37.654125 19177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:43:37.688180 19177 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:43:37.689600 19177 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:43:37.700958 19177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977: Generated new TSK 0
I20260924 10:43:37.701877 19177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:43:37.734138 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:42041
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:37.846647 19182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:37.846932 19182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:37.847016 19182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:37.851172 19182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:37.851238 19182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:37.851351 19182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:43:37.856832 19182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:42041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:37.858067 19182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:37.859225 19182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:37.869680 19188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:37.869733 19187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:37.869748 19190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:37.870050 19182 server_base.cc:1034] running on GCE node
I20260924 10:43:37.870430 19182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:37.871093 19182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:37.872260 19182 hybrid_clock.cc:648] HybridClock initialized: now 1790246617872227 us; error 45 us; skew 500 ppm
Sep 24 10:43:37 dist-test-slave-j38h krb5kdc[19075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246617, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:37.875226 19182 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:43:37.876152 19182 webserver.cc:460] Webserver started at http://127.2.8.1:40405/ using document root <none> and password file <none>
I20260924 10:43:37.876732 19182 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:37.876830 19182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:37.877142 19182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:37.879206 19182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "37b726eb0bfd40c6817435510e24a004"
format_stamp: "Formatted at 2026-09-24 10:43:37 on dist-test-slave-j38h"
server_key: "b7912e513cfa37b7bd1af971504f95a2"
server_key_iv: "295250a0cda0ea25a0b69e623ccddc70"
server_key_version: "encryptionkey@0"
I20260924 10:43:37.879786 19182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "37b726eb0bfd40c6817435510e24a004"
format_stamp: "Formatted at 2026-09-24 10:43:37 on dist-test-slave-j38h"
server_key: "b7912e513cfa37b7bd1af971504f95a2"
server_key_iv: "295250a0cda0ea25a0b69e623ccddc70"
server_key_version: "encryptionkey@0"
I20260924 10:43:37.883739 19182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:43:37.886449 19197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:37.887686 19182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:43:37.887825 19182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "37b726eb0bfd40c6817435510e24a004"
format_stamp: "Formatted at 2026-09-24 10:43:37 on dist-test-slave-j38h"
server_key: "b7912e513cfa37b7bd1af971504f95a2"
server_key_iv: "295250a0cda0ea25a0b69e623ccddc70"
server_key_version: "encryptionkey@0"
I20260924 10:43:37.887943 19182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:37.905673 19182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:37.909307 19182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:37.909523 19182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:37.910136 19182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:37.911060 19182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:37.911136 19182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:37.911204 19182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:37.911255 19182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:37.920523 19182 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:33709
I20260924 10:43:37.920554 19310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:33709 every 8 connection(s)
I20260924 10:43:37.921581 19182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
Sep 24 10:43:37 dist-test-slave-j38h krb5kdc[19075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246617, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:43:37.931001 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 19182
I20260924 10:43:37.931110 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:43:37.931428 2080 external_mini_cluster.cc:1428] Setting key 9dbb047b16d01d9d9730d35b7a65bf88
I20260924 10:43:37.935437 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:37.923483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:48189 (local address 127.2.8.62:42041)
0924 10:43:37.923655 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:37.923659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:37.924745 (+ 1086us) server_negotiation.cc:407] Connection header received
0924 10:43:37.925678 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:37.925682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:37.925739 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:37.925859 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:37.927037 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:37.927602 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:37.928776 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:37.929086 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:37.929201 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:37.931723 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:37.931726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:37.931728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:37.931754 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:37.933308 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:37.933882 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:37.933887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:37.933889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:37.933977 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:37.934332 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:37.934337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:37.934339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:37.934523 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:37.934631 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:37.935146 (+ 515us) server_negotiation.cc:299] Negotiation successful
0924 10:43:37.935274 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 10:43:37.936336 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:37.923834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:42041 (local address 127.2.8.1:48189)
0924 10:43:37.924566 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:37.924616 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:37.925464 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:37.926034 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:37.926041 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:37.926405 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:37.926871 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:37.926884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:37.927743 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:37.927747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:37.928632 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:37.928641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:37.929188 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:37.929245 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:37.929441 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:37.929979 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:43:37.929994 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:37.931536 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:37.933483 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:37.933489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:37.933504 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:37.933766 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:37.934075 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:37.934079 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:37.934081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:37.934228 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:37.934642 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:37.934650 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:37.934903 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 10:43:37.935105 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 10:43:37.935427 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":129,"threads_started":1}
I20260924 10:43:37.937685 19311 heartbeater.cc:347] Connected to a master server at 127.2.8.62:42041
I20260924 10:43:37.937922 19311 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:37.938418 19311 heartbeater.cc:511] Master 127.2.8.62:42041 requested a full tablet report, sending...
I20260924 10:43:37.939966 19120 ts_manager.cc:194] Registered new tserver with Master: 37b726eb0bfd40c6817435510e24a004 (127.2.8.1:33709)
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
I20260924 10:43:37.941231 19120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:48189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:43:37.960631 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:42041
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:38.071987 19316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:38.072268 19316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:38.072312 19316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:38.076323 19316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:38.076457 19316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:38.076540 19316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:43:38.081988 19316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:42041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:38.083102 19316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:38.084154 19316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:38.093910 19324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:38.093941 19321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:38.093931 19322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:38.094548 19316 server_base.cc:1034] running on GCE node
I20260924 10:43:38.095075 19316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:38.095752 19316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:38.097084 19316 hybrid_clock.cc:648] HybridClock initialized: now 1790246618097048 us; error 40 us; skew 500 ppm
Sep 24 10:43:38 dist-test-slave-j38h krb5kdc[19075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246618, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:38.100118 19316 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:43:38.101178 19316 webserver.cc:460] Webserver started at http://127.2.8.2:33101/ using document root <none> and password file <none>
I20260924 10:43:38.101747 19316 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:38.101848 19316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:38.102061 19316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:38.104099 19316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "4417797329d34402b593bd4ad645b42b"
format_stamp: "Formatted at 2026-09-24 10:43:38 on dist-test-slave-j38h"
server_key: "32f50222b7acfc9246f84e7a3217b64c"
server_key_iv: "62194f960ddf2d2f12e8d080b582a2a4"
server_key_version: "encryptionkey@0"
I20260924 10:43:38.104672 19316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "4417797329d34402b593bd4ad645b42b"
format_stamp: "Formatted at 2026-09-24 10:43:38 on dist-test-slave-j38h"
server_key: "32f50222b7acfc9246f84e7a3217b64c"
server_key_iv: "62194f960ddf2d2f12e8d080b582a2a4"
server_key_version: "encryptionkey@0"
I20260924 10:43:38.108516 19316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:43:38.111202 19331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:38.112493 19316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:43:38.112677 19316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "4417797329d34402b593bd4ad645b42b"
format_stamp: "Formatted at 2026-09-24 10:43:38 on dist-test-slave-j38h"
server_key: "32f50222b7acfc9246f84e7a3217b64c"
server_key_iv: "62194f960ddf2d2f12e8d080b582a2a4"
server_key_version: "encryptionkey@0"
I20260924 10:43:38.112809 19316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:38.128134 19316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:38.131739 19316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:38.131964 19316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:38.132582 19316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:38.133518 19316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:38.133597 19316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:38.133661 19316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:38.133697 19316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:38.143160 19316 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:33993
I20260924 10:43:38.143186 19444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:33993 every 8 connection(s)
I20260924 10:43:38.144330 19316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:43:38.147044 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 19316
I20260924 10:43:38.147197 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:43:38.147542 2080 external_mini_cluster.cc:1428] Setting key 18df28089d86d6b86cd26450183d9c66
Sep 24 10:43:38 dist-test-slave-j38h krb5kdc[19075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246618, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
I20260924 10:43:38.160158 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.146365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:46575 (local address 127.2.8.62:42041)
0924 10:43:38.146530 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:38.146533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:38.147509 (+ 976us) server_negotiation.cc:407] Connection header received
0924 10:43:38.148627 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:38.148630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:38.148673 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:38.148744 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:38.150027 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.150877 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.152833 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.153238 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.153322 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:38.156317 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:38.156321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:38.156323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:38.156354 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:38.158050 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:38.158587 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:38.158591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:38.158593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:38.158673 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:38.159115 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:38.159117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:38.159119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:38.159313 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:38.159397 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:38.159910 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 10:43:38.160038 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
I20260924 10:43:38.161206 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.146736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:42041 (local address 127.2.8.2:46575)
0924 10:43:38.147286 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:38.147367 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:38.148410 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:38.148921 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:38.148930 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:38.149381 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:38.149874 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:38.149887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.151031 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.151036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:38.152686 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:38.152700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.153357 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.153417 (+ 60us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:38.153650 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:38.154328 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:43:38.154354 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:38.156132 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:38.158177 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:38.158184 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:38.158200 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:38.158469 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:38.158778 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:38.158781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:38.158783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:38.159008 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:38.159415 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:38.159420 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:38.159678 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 10:43:38.159843 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 10:43:38.160131 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":153,"threads_started":1}
I20260924 10:43:38.162504 19445 heartbeater.cc:347] Connected to a master server at 127.2.8.62:42041
I20260924 10:43:38.162766 19445 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:38.163393 19445 heartbeater.cc:511] Master 127.2.8.62:42041 requested a full tablet report, sending...
I20260924 10:43:38.164594 19120 ts_manager.cc:194] Registered new tserver with Master: 4417797329d34402b593bd4ad645b42b (127.2.8.2:33993)
I20260924 10:43:38.165278 19120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:46575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:43:38.182052 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:42041
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:38.294376 19450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:38.294678 19450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:38.294754 19450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:38.298983 19450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:38.299062 19450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:38.299180 19450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:43:38.304648 19450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:42041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:38.305784 19450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:38.306829 19450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:38.316864 19458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:38.316866 19456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:38.316866 19455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:38.317317 19450 server_base.cc:1034] running on GCE node
I20260924 10:43:38.317782 19450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:38.318508 19450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:38.319706 19450 hybrid_clock.cc:648] HybridClock initialized: now 1790246618319687 us; error 31 us; skew 500 ppm
Sep 24 10:43:38 dist-test-slave-j38h krb5kdc[19075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246618, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:38.322894 19450 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:43:38.323789 19450 webserver.cc:460] Webserver started at http://127.2.8.3:37539/ using document root <none> and password file <none>
I20260924 10:43:38.324376 19450 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:38.324441 19450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:38.324700 19450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:38.326824 19450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "641233a5990944078b54bf195585b71e"
format_stamp: "Formatted at 2026-09-24 10:43:38 on dist-test-slave-j38h"
server_key: "35662335592af5b5ac9e133263001b2d"
server_key_iv: "2a7eef7f4eb0befa66072cb191d7cd04"
server_key_version: "encryptionkey@0"
I20260924 10:43:38.327417 19450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "641233a5990944078b54bf195585b71e"
format_stamp: "Formatted at 2026-09-24 10:43:38 on dist-test-slave-j38h"
server_key: "35662335592af5b5ac9e133263001b2d"
server_key_iv: "2a7eef7f4eb0befa66072cb191d7cd04"
server_key_version: "encryptionkey@0"
I20260924 10:43:38.331297 19450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 10:43:38.333940 19465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:38.335208 19450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:43:38.335397 19450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "641233a5990944078b54bf195585b71e"
format_stamp: "Formatted at 2026-09-24 10:43:38 on dist-test-slave-j38h"
server_key: "35662335592af5b5ac9e133263001b2d"
server_key_iv: "2a7eef7f4eb0befa66072cb191d7cd04"
server_key_version: "encryptionkey@0"
I20260924 10:43:38.335522 19450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:38.350910 19450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:38.354452 19450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:38.354699 19450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:38.355340 19450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:38.356171 19450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:38.356217 19450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:38.356295 19450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:38.356336 19450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:38.365530 19450 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:37443
I20260924 10:43:38.365553 19578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:37443 every 8 connection(s)
I20260924 10:43:38.366451 19450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:43:38.368585 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 19450
I20260924 10:43:38.368721 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:43:38.369112 2080 external_mini_cluster.cc:1428] Setting key 1f4c091f7300df9f86b43918492a3107
Sep 24 10:43:38 dist-test-slave-j38h krb5kdc[19075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246618, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:43:38.380219 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.368328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:48597 (local address 127.2.8.62:42041)
0924 10:43:38.368475 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:38.368480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:38.369605 (+ 1125us) server_negotiation.cc:407] Connection header received
0924 10:43:38.370809 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:38.370812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:38.370865 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:38.370957 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:38.372314 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.372847 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.374085 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.374292 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.374377 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:38.376829 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:38.376833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:38.376835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:38.376859 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:38.378446 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:38.378887 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:38.378889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:38.378890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:38.378960 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:38.379252 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:38.379254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:38.379256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:38.379462 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:38.379541 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:38.379986 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 10:43:38.380099 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:43:38.381158 19581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.368789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:42041 (local address 127.2.8.3:48597)
0924 10:43:38.369365 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:38.369409 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:38.370556 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:38.371128 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:38.371134 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:38.371671 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:38.372164 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:38.372178 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.373020 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.373026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:38.373963 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:38.373972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.374388 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.374430 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:38.374569 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:38.375103 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:43:38.375119 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:38.376668 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:38.378541 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:38.378546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:38.378557 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:38.378797 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:38.379041 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:38.379043 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:38.379045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:38.379169 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:38.379554 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:38.379558 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:38.379782 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 10:43:38.379936 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 10:43:38.380203 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":177,"threads_started":1}
I20260924 10:43:38.382426 19579 heartbeater.cc:347] Connected to a master server at 127.2.8.62:42041
I20260924 10:43:38.382725 19579 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:38.383315 19579 heartbeater.cc:511] Master 127.2.8.62:42041 requested a full tablet report, sending...
I20260924 10:43:38.384310 19120 ts_manager.cc:194] Registered new tserver with Master: 641233a5990944078b54bf195585b71e (127.2.8.3:37443)
I20260924 10:43:38.384845 19120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:48597
I20260924 10:43:38.395390 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:43:38.405324 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.397520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.2.8.62:42041)
0924 10:43:38.397668 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:38.397672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:38.397774 (+ 102us) server_negotiation.cc:407] Connection header received
0924 10:43:38.397856 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:38.397859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:38.397905 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:38.397982 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:38.398690 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.399271 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.400175 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.400400 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.400488 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:38.401932 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:38.401936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:38.401938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:38.401964 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:38.403628 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:38.404021 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:38.404023 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:38.404025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:38.404108 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:38.404354 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:38.404356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:38.404357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:38.404586 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:38.404701 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:38.404901 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 10:43:38.405083 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 10:43:38.408320 19120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:43:38.411429 19120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:43:38.427613 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.422182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39488 (local address 127.2.8.3:37443)
0924 10:43:38.422775 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:38.422789 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:38.423106 (+ 317us) server_negotiation.cc:407] Connection header received
0924 10:43:38.423185 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:38.423192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:38.423369 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:38.423526 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:38.423742 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.424830 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.426264 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.426820 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.426969 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:38.427113 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:38.427327 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 10:43:38.427442 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":90,"threads_started":1}
I20260924 10:43:38.427606 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.421779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:37443 (local address 127.0.0.1:39488)
0924 10:43:38.422905 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:38.422918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:38.423012 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:38.423539 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:38.423542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:38.423561 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:38.423645 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:38.423651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.425034 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.425037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:38.426171 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:38.426179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.426976 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.426979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:38.427109 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:38.427153 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 10:43:38.427294 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 10:43:38.427440 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"mutex_wait_us":42,"thread_start_us":94,"threads_started":1}
I20260924 10:43:38.427948 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.421160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36470 (local address 127.2.8.2:33993)
0924 10:43:38.422116 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:38.422130 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:38.422153 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:43:38.422234 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:38.422243 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:38.422487 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:38.422652 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:38.423157 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.424343 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.426229 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.427005 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.427111 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:38.427242 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:38.427674 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 10:43:38.427777 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":101,"threads_started":1}
I20260924 10:43:38.428001 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.421039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:33993 (local address 127.0.0.1:36470)
0924 10:43:38.421642 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:38.421694 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:38.421812 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:38.422851 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:38.422857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:38.422881 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:38.423030 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:38.423040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.424501 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.424504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:38.426079 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:38.426089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.427300 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.427304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:38.427384 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:38.427405 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:43:38.427479 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 10:43:38.427555 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":114,"thread_start_us":88,"threads_started":1}
I20260924 10:43:38.428398 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.421987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56344 (local address 127.2.8.1:33709)
0924 10:43:38.422332 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:38.422349 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:38.422649 (+ 300us) server_negotiation.cc:407] Connection header received
0924 10:43:38.422734 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:38.422737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:38.422871 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:38.423042 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:38.423239 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.424401 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.425911 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.426723 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.426834 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:38.426988 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:38.428195 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0924 10:43:38.428275 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":183,"threads_started":1}
I20260924 10:43:38.428719 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.421114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:33709 (local address 127.0.0.1:56344)
0924 10:43:38.422440 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:38.422451 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:38.422569 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:38.423021 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:38.423024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:38.423042 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:38.423142 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:38.423148 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.424542 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.424545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:38.425765 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:38.425774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.427058 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.427064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:38.427184 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:38.427206 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:43:38.427279 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 10:43:38.428632 (+ 1353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1250,"mutex_wait_us":132,"thread_start_us":105,"threads_started":1}
I20260924 10:43:38.430886 19379 tablet_service.cc:1467] Processing CreateTablet for tablet 58290fc1384a464bb56b1c54a1feb777 (DEFAULT_TABLE table=test-table [id=5cbe196eabb24e69bb2ec77fca69a615]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:38.431485 19513 tablet_service.cc:1467] Processing CreateTablet for tablet 58290fc1384a464bb56b1c54a1feb777 (DEFAULT_TABLE table=test-table [id=5cbe196eabb24e69bb2ec77fca69a615]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:38.431484 19245 tablet_service.cc:1467] Processing CreateTablet for tablet 58290fc1384a464bb56b1c54a1feb777 (DEFAULT_TABLE table=test-table [id=5cbe196eabb24e69bb2ec77fca69a615]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:38.432014 19379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58290fc1384a464bb56b1c54a1feb777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:38.432469 19245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58290fc1384a464bb56b1c54a1feb777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:38.432469 19513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58290fc1384a464bb56b1c54a1feb777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:38.438206 19596 tablet_bootstrap.cc:492] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b: Bootstrap starting.
I20260924 10:43:38.439723 19597 tablet_bootstrap.cc:492] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004: Bootstrap starting.
I20260924 10:43:38.441351 19598 tablet_bootstrap.cc:492] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e: Bootstrap starting.
I20260924 10:43:38.441605 19596 tablet_bootstrap.cc:654] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:38.441913 19597 tablet_bootstrap.cc:654] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:38.442538 19596 log.cc:824] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:38.442667 19597 log.cc:824] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:38.443683 19598 tablet_bootstrap.cc:654] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:38.444664 19596 tablet_bootstrap.cc:492] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b: No bootstrap required, opened a new log
I20260924 10:43:38.444828 19598 log.cc:824] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:38.444893 19596 ts_tablet_manager.cc:1397] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 10:43:38.446553 19598 tablet_bootstrap.cc:492] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e: No bootstrap required, opened a new log
I20260924 10:43:38.446576 19597 tablet_bootstrap.cc:492] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004: No bootstrap required, opened a new log
I20260924 10:43:38.446760 19597 ts_tablet_manager.cc:1397] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 10:43:38.446774 19598 ts_tablet_manager.cc:1397] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:43:38.448952 19596 raft_consensus.cc:348] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4417797329d34402b593bd4ad645b42b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33993 } } peers { permanent_uuid: "37b726eb0bfd40c6817435510e24a004" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33709 } } peers { permanent_uuid: "641233a5990944078b54bf195585b71e" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37443 } }
I20260924 10:43:38.449261 19596 raft_consensus.cc:374] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:38.449318 19596 raft_consensus.cc:729] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4417797329d34402b593bd4ad645b42b, State: Initialized, Role: FOLLOWER
I20260924 10:43:38.449787 19596 consensus_queue.cc:260] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4417797329d34402b593bd4ad645b42b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33993 } } peers { permanent_uuid: "37b726eb0bfd40c6817435510e24a004" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33709 } } peers { permanent_uuid: "641233a5990944078b54bf195585b71e" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37443 } }
I20260924 10:43:38.449982 19598 raft_consensus.cc:348] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4417797329d34402b593bd4ad645b42b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33993 } } peers { permanent_uuid: "37b726eb0bfd40c6817435510e24a004" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33709 } } peers { permanent_uuid: "641233a5990944078b54bf195585b71e" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37443 } }
I20260924 10:43:38.450270 19598 raft_consensus.cc:374] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:38.450335 19598 raft_consensus.cc:729] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 641233a5990944078b54bf195585b71e, State: Initialized, Role: FOLLOWER
I20260924 10:43:38.450604 19445 heartbeater.cc:503] Master 127.2.8.62:42041 was elected leader, sending a full tablet report...
I20260924 10:43:38.450796 19596 ts_tablet_manager.cc:1428] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 10:43:38.450829 19598 consensus_queue.cc:260] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4417797329d34402b593bd4ad645b42b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33993 } } peers { permanent_uuid: "37b726eb0bfd40c6817435510e24a004" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33709 } } peers { permanent_uuid: "641233a5990944078b54bf195585b71e" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37443 } }
I20260924 10:43:38.451560 19579 heartbeater.cc:503] Master 127.2.8.62:42041 was elected leader, sending a full tablet report...
I20260924 10:43:38.451815 19598 ts_tablet_manager.cc:1428] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:43:38.452318 19597 raft_consensus.cc:348] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4417797329d34402b593bd4ad645b42b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33993 } } peers { permanent_uuid: "37b726eb0bfd40c6817435510e24a004" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33709 } } peers { permanent_uuid: "641233a5990944078b54bf195585b71e" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37443 } }
I20260924 10:43:38.452574 19597 raft_consensus.cc:374] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:38.452636 19597 raft_consensus.cc:729] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b726eb0bfd40c6817435510e24a004, State: Initialized, Role: FOLLOWER
I20260924 10:43:38.453085 19597 consensus_queue.cc:260] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4417797329d34402b593bd4ad645b42b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33993 } } peers { permanent_uuid: "37b726eb0bfd40c6817435510e24a004" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33709 } } peers { permanent_uuid: "641233a5990944078b54bf195585b71e" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37443 } }
I20260924 10:43:38.453723 19311 heartbeater.cc:503] Master 127.2.8.62:42041 was elected leader, sending a full tablet report...
I20260924 10:43:38.454079 19597 ts_tablet_manager.cc:1428] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260924 10:43:38.617573 19580 tablet.cc:2369] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:43:38.645659 19446 tablet.cc:2369] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 10:43:38.672935 19312 tablet.cc:2369] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:43:38.714195 19604 raft_consensus.cc:482] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:43:38.714381 19604 raft_consensus.cc:504] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4417797329d34402b593bd4ad645b42b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33993 } } peers { permanent_uuid: "37b726eb0bfd40c6817435510e24a004" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33709 } } peers { permanent_uuid: "641233a5990944078b54bf195585b71e" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37443 } }
I20260924 10:43:38.715408 19604 leader_election.cc:288] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4417797329d34402b593bd4ad645b42b (127.2.8.2:33993), 641233a5990944078b54bf195585b71e (127.2.8.3:37443)
I20260924 10:43:38.719601 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.715838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:56287 (local address 127.2.8.2:33993)
0924 10:43:38.715980 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:38.715986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:38.716238 (+ 252us) server_negotiation.cc:407] Connection header received
0924 10:43:38.716487 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:38.716490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:38.716546 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:38.716652 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:38.716944 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.717563 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.718640 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.719041 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.719168 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:38.719236 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:38.719413 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 10:43:38.719482 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:43:38.719619 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.715755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:33993 (local address 127.2.8.1:56287)
0924 10:43:38.716095 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:38.716113 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:38.716244 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:38.716666 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:38.716669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:38.716685 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:38.716824 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:38.716830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.717683 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.717687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:38.718508 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:38.718517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.719171 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.719174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:38.719272 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:38.719295 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:43:38.719399 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 10:43:38.719471 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":130,"threads_started":1}
I20260924 10:43:38.720039 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.716368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:47445 (local address 127.2.8.3:37443)
0924 10:43:38.716949 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:38.716954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:38.716968 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:43:38.717074 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:38.717076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:38.717114 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:38.717204 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:38.717479 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.717962 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.719042 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.719443 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.719562 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:38.719616 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:38.719839 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 10:43:38.719901 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468}
I20260924 10:43:38.720371 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.716203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:37443 (local address 127.2.8.1:47445)
0924 10:43:38.716614 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:38.716679 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:38.716771 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:38.717224 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:38.717227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:38.717243 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:38.717368 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:38.717373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.718084 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.718089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:38.718902 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:38.718911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.719575 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.719579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:38.719681 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:38.719716 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 10:43:38.720202 (+ 486us) client_negotiation.cc:240] Negotiation successful
0924 10:43:38.720272 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":143,"threads_started":1}
I20260924 10:43:38.720300 19399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58290fc1384a464bb56b1c54a1feb777" candidate_uuid: "37b726eb0bfd40c6817435510e24a004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4417797329d34402b593bd4ad645b42b" is_pre_election: true
I20260924 10:43:38.720651 19399 raft_consensus.cc:2454] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37b726eb0bfd40c6817435510e24a004 in term 0.
I20260924 10:43:38.721055 19533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58290fc1384a464bb56b1c54a1feb777" candidate_uuid: "37b726eb0bfd40c6817435510e24a004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "641233a5990944078b54bf195585b71e" is_pre_election: true
I20260924 10:43:38.721194 19201 leader_election.cc:302] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37b726eb0bfd40c6817435510e24a004, 4417797329d34402b593bd4ad645b42b; no voters:
I20260924 10:43:38.721344 19533 raft_consensus.cc:2454] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37b726eb0bfd40c6817435510e24a004 in term 0.
I20260924 10:43:38.721498 19604 raft_consensus.cc:2781] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:43:38.721621 19604 raft_consensus.cc:482] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:43:38.721683 19604 raft_consensus.cc:3037] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:38.722848 19604 raft_consensus.cc:504] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4417797329d34402b593bd4ad645b42b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33993 } } peers { permanent_uuid: "37b726eb0bfd40c6817435510e24a004" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33709 } } peers { permanent_uuid: "641233a5990944078b54bf195585b71e" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37443 } }
I20260924 10:43:38.723212 19604 leader_election.cc:288] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [CANDIDATE]: Term 1 election: Requested vote from peers 4417797329d34402b593bd4ad645b42b (127.2.8.2:33993), 641233a5990944078b54bf195585b71e (127.2.8.3:37443)
I20260924 10:43:38.723584 19399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58290fc1384a464bb56b1c54a1feb777" candidate_uuid: "37b726eb0bfd40c6817435510e24a004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4417797329d34402b593bd4ad645b42b"
I20260924 10:43:38.723680 19533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58290fc1384a464bb56b1c54a1feb777" candidate_uuid: "37b726eb0bfd40c6817435510e24a004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "641233a5990944078b54bf195585b71e"
I20260924 10:43:38.723740 19399 raft_consensus.cc:3037] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:38.723850 19533 raft_consensus.cc:3037] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:38.724792 19399 raft_consensus.cc:2454] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37b726eb0bfd40c6817435510e24a004 in term 1.
I20260924 10:43:38.725015 19533 raft_consensus.cc:2454] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37b726eb0bfd40c6817435510e24a004 in term 1.
I20260924 10:43:38.725167 19201 leader_election.cc:302] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37b726eb0bfd40c6817435510e24a004, 4417797329d34402b593bd4ad645b42b; no voters:
I20260924 10:43:38.725374 19604 raft_consensus.cc:2781] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:38.725627 19604 raft_consensus.cc:686] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [term 1 LEADER]: Becoming Leader. State: Replica: 37b726eb0bfd40c6817435510e24a004, State: Running, Role: LEADER
I20260924 10:43:38.725951 19604 consensus_queue.cc:237] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4417797329d34402b593bd4ad645b42b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33993 } } peers { permanent_uuid: "37b726eb0bfd40c6817435510e24a004" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33709 } } peers { permanent_uuid: "641233a5990944078b54bf195585b71e" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37443 } }
I20260924 10:43:38.729158 19118 catalog_manager.cc:5495] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37b726eb0bfd40c6817435510e24a004 (127.2.8.1). New cstate: current_term: 1 leader_uuid: "37b726eb0bfd40c6817435510e24a004" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4417797329d34402b593bd4ad645b42b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37b726eb0bfd40c6817435510e24a004" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 33709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "641233a5990944078b54bf195585b71e" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37443 } health_report { overall_health: UNKNOWN } } }
I20260924 10:43:38.790983 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.787506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56354 (local address 127.2.8.1:33709)
0924 10:43:38.787660 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:38.787664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:38.787677 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:43:38.787759 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:38.787762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:38.787815 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:38.787904 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:43:38.788154 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.788663 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.789553 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:38.789761 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:38.789853 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:38.790123 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:43:38.790547 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:43:38.790639 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:38.790799 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 10:43:38.790852 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 10:43:38.797364 19533 raft_consensus.cc:1261] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e [term 1 FOLLOWER]: Refusing update from remote peer 37b726eb0bfd40c6817435510e24a004: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:43:38.797364 19399 raft_consensus.cc:1261] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b [term 1 FOLLOWER]: Refusing update from remote peer 37b726eb0bfd40c6817435510e24a004: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:43:38.798137 19607 consensus_queue.cc:1035] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4417797329d34402b593bd4ad645b42b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:43:38.798473 19604 consensus_queue.cc:1035] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004 [LEADER]: Connected to new peer: Peer: permanent_uuid: "641233a5990944078b54bf195585b71e" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:43:38.805791 19610 mvcc.cc:205] Tried to move back new op lower bound from 7332850150585331712 to 7332850150303817728. Current Snapshot: MvccSnapshot[applied={T|T < 7332850150585331712}]
I20260924 10:43:38.807124 19612 mvcc.cc:205] Tried to move back new op lower bound from 7332850150585331712 to 7332850150303817728. Current Snapshot: MvccSnapshot[applied={T|T < 7332850150585331712}]
I20260924 10:43:38.807354 19611 mvcc.cc:205] Tried to move back new op lower bound from 7332850150585331712 to 7332850150303817728. Current Snapshot: MvccSnapshot[applied={T|T < 7332850150585331712}]
I20260924 10:43:38.814555 19118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:43:38.814713 19118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:43:38.817291 19118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71fa946349ad47e9ac86f74946f42977: Sending DeleteTablet for 3 replicas of tablet 58290fc1384a464bb56b1c54a1feb777
I20260924 10:43:38.818113 19245 tablet_service.cc:1514] Processing DeleteTablet for tablet 58290fc1384a464bb56b1c54a1feb777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:38 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:56344
I20260924 10:43:38.818142 19513 tablet_service.cc:1514] Processing DeleteTablet for tablet 58290fc1384a464bb56b1c54a1feb777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:38 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:39488
I20260924 10:43:38.818147 19379 tablet_service.cc:1514] Processing DeleteTablet for tablet 58290fc1384a464bb56b1c54a1feb777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:38 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:36470
I20260924 10:43:38.818645 19621 tablet_replica.cc:330] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b: stopping tablet replica
I20260924 10:43:38.818650 19620 tablet_replica.cc:330] T 58290fc1384a464bb56b1c54a1feb777 P 37b726eb0bfd40c6817435510e24a004: stopping tablet replica
I20260924 10:43:38.818989 19621 raft_consensus.cc:2229] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:43:38.819150 19622 tablet_replica.cc:330] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e: stopping tablet replica
I20260924 10:43:38.819278 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 19182
I20260924 10:43:38.819319 19621 raft_consensus.cc:2258] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:43:38.819397 19622 raft_consensus.cc:2229] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:43:38.819933 19622 raft_consensus.cc:2258] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:43:38.821683 19621 ts_tablet_manager.cc:1905] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:43:38.821921 19622 ts_tablet_manager.cc:1905] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:43:38.824431 19621 ts_tablet_manager.cc:1918] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:43:38.824431 19622 ts_tablet_manager.cc:1918] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:43:38.824535 19622 log.cc:1196] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/58290fc1384a464bb56b1c54a1feb777
I20260924 10:43:38.824535 19621 log.cc:1196] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/58290fc1384a464bb56b1c54a1feb777
I20260924 10:43:38.824857 19622 ts_tablet_manager.cc:1939] T 58290fc1384a464bb56b1c54a1feb777 P 641233a5990944078b54bf195585b71e: Deleting consensus metadata
I20260924 10:43:38.824857 19621 ts_tablet_manager.cc:1939] T 58290fc1384a464bb56b1c54a1feb777 P 4417797329d34402b593bd4ad645b42b: Deleting consensus metadata
I20260924 10:43:38.826390 19107 catalog_manager.cc:4841] TS 641233a5990944078b54bf195585b71e (127.2.8.3:37443): tablet 58290fc1384a464bb56b1c54a1feb777 (table test-table [id=5cbe196eabb24e69bb2ec77fca69a615]) successfully deleted
W20260924 10:43:38.826512 19104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.8.1:33709 (error 108)
I20260924 10:43:38.826598 19107 catalog_manager.cc:4841] TS 4417797329d34402b593bd4ad645b42b (127.2.8.2:33993): tablet 58290fc1384a464bb56b1c54a1feb777 (table test-table [id=5cbe196eabb24e69bb2ec77fca69a615]) successfully deleted
I20260924 10:43:38.826783 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 19316
I20260924 10:43:38.828032 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:38.827666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:33709 (local address 127.0.0.1:56364)
0924 10:43:38.827834 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:38.827918 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:33709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 10:43:38.828258 19104 catalog_manager.cc:4584] TS 37b726eb0bfd40c6817435510e24a004 (127.2.8.1:33709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 58290fc1384a464bb56b1c54a1feb777: Network error: Client connection negotiation failed: client connection to 127.2.8.1:33709: connect: Connection refused (error 111)
I20260924 10:43:38.832515 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 19450
I20260924 10:43:38.837962 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 19088
2026-09-24T10:43:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2577 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T10:43:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:43:38Z Disabled control of system clock
I20260924 10:43:38.869637 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:37021
--webserver_interface=127.2.8.62
--webserver_port=0
--builtin_ntp_servers=127.2.8.20:46663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:37021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:43:38.984156 19627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:38.984460 19627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:38.984499 19627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:38.988727 19627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:43:38.988814 19627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:38.988831 19627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:38.988857 19627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:43:38.988873 19627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:43:38.993988 19627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:46663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:37021
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:37021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:38.995110 19627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:38.996587 19627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:39.005331 19635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:39.005380 19633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:39.005465 19627 server_base.cc:1034] running on GCE node
W20260924 10:43:39.005350 19632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:39.006146 19627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:39.007294 19627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:39.008517 19627 hybrid_clock.cc:648] HybridClock initialized: now 1790246619008487 us; error 45 us; skew 500 ppm
I20260924 10:43:39.010537 19627 webserver.cc:460] Webserver started at http://127.2.8.62:37331/ using document root <none> and password file <none>
I20260924 10:43:39.011134 19627 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:39.011234 19627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:39.011461 19627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:39.013677 19627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "b9c28d2ae9784d7f8c44fa7d156c1414"
format_stamp: "Formatted at 2026-09-24 10:43:39 on dist-test-slave-j38h"
server_key: "f1adf4d6d53eb9b600a36e49cd75d251"
server_key_iv: "27f69bcc02eea5cbb14dc30968c0fdee"
server_key_version: "encryptionkey@0"
I20260924 10:43:39.014266 19627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "b9c28d2ae9784d7f8c44fa7d156c1414"
format_stamp: "Formatted at 2026-09-24 10:43:39 on dist-test-slave-j38h"
server_key: "f1adf4d6d53eb9b600a36e49cd75d251"
server_key_iv: "27f69bcc02eea5cbb14dc30968c0fdee"
server_key_version: "encryptionkey@0"
I20260924 10:43:39.018115 19627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:43:39.020581 19641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:39.021890 19627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:43:39.022042 19627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "b9c28d2ae9784d7f8c44fa7d156c1414"
format_stamp: "Formatted at 2026-09-24 10:43:39 on dist-test-slave-j38h"
server_key: "f1adf4d6d53eb9b600a36e49cd75d251"
server_key_iv: "27f69bcc02eea5cbb14dc30968c0fdee"
server_key_version: "encryptionkey@0"
I20260924 10:43:39.022159 19627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:39.034334 19627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:39.035018 19627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:39.035223 19627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:39.042517 19627 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:37021
I20260924 10:43:39.042533 19693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:37021 every 8 connection(s)
I20260924 10:43:39.043478 19627 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:43:39.046175 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 19627
I20260924 10:43:39.046334 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:43:39.046636 2080 external_mini_cluster.cc:1428] Setting key db87defcff14939c2a894463e75ff87b
I20260924 10:43:39.046676 19694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:39.053293 19694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414: Bootstrap starting.
I20260924 10:43:39.055588 19694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:39.055753 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.047957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.2.8.62:37021)
0924 10:43:39.048549 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:39.048564 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:39.048599 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:43:39.049328 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:39.049346 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:39.049739 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:39.050099 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:39.050534 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.051480 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.052388 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.052870 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.053082 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.054013 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:39.054018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:39.054031 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:39.054412 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:39.054475 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:39.054495 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:39.054612 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:39.054732 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 10:43:39.054994 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":113,"threads_started":1}
I20260924 10:43:39.056406 19694 log.cc:824] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:39.058459 19694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414: No bootstrap required, opened a new log
I20260924 10:43:39.062232 19694 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c28d2ae9784d7f8c44fa7d156c1414" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 37021 } }
I20260924 10:43:39.062454 19694 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:39.062536 19694 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c28d2ae9784d7f8c44fa7d156c1414, State: Initialized, Role: FOLLOWER
I20260924 10:43:39.062968 19694 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c28d2ae9784d7f8c44fa7d156c1414" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 37021 } }
I20260924 10:43:39.063090 19694 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:43:39.063186 19694 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:43:39.063285 19694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:39.064251 19694 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c28d2ae9784d7f8c44fa7d156c1414" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 37021 } }
I20260924 10:43:39.064603 19694 leader_election.cc:302] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9c28d2ae9784d7f8c44fa7d156c1414; no voters:
I20260924 10:43:39.064919 19694 leader_election.cc:288] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:43:39.065084 19699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:39.065336 19699 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [term 1 LEADER]: Becoming Leader. State: Replica: b9c28d2ae9784d7f8c44fa7d156c1414, State: Running, Role: LEADER
I20260924 10:43:39.065644 19699 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c28d2ae9784d7f8c44fa7d156c1414" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 37021 } }
I20260924 10:43:39.066299 19694 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:43:39.067515 19701 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9c28d2ae9784d7f8c44fa7d156c1414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c28d2ae9784d7f8c44fa7d156c1414" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 37021 } } }
I20260924 10:43:39.067512 19700 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9c28d2ae9784d7f8c44fa7d156c1414. Latest consensus state: current_term: 1 leader_uuid: "b9c28d2ae9784d7f8c44fa7d156c1414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9c28d2ae9784d7f8c44fa7d156c1414" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 37021 } } }
I20260924 10:43:39.067659 19701 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:39.067668 19700 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414 [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:39.068074 19708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:43:39.070941 19708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:43:39.077045 19708 catalog_manager.cc:1347] Generated new cluster ID: f14c9451e9074d11b104941dfcddf210
I20260924 10:43:39.077112 19708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:43:39.110854 19708 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:43:39.111863 19708 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:43:39.118124 19708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414: Generated new TSK 0
I20260924 10:43:39.118839 19708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:43:39.132697 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:37021
--builtin_ntp_servers=127.2.8.20:46663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:43:39.247538 19718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:39.247807 19718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:39.247845 19718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:39.252094 19718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:39.252162 19718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:39.252261 19718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:43:39.257215 19718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:46663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:37021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:39.258397 19718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:39.259814 19718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:39.269810 19724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:39.269824 19726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:39.269887 19718 server_base.cc:1034] running on GCE node
W20260924 10:43:39.269815 19723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:39.270440 19718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:39.271121 19718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:39.272317 19718 hybrid_clock.cc:648] HybridClock initialized: now 1790246619272284 us; error 44 us; skew 500 ppm
I20260924 10:43:39.274461 19718 webserver.cc:460] Webserver started at http://127.2.8.1:44409/ using document root <none> and password file <none>
I20260924 10:43:39.275067 19718 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:39.275130 19718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:39.275394 19718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:39.277586 19718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "9168d0d358574e1eaf41c3136d7fdb62"
format_stamp: "Formatted at 2026-09-24 10:43:39 on dist-test-slave-j38h"
server_key: "4662ddb715b93d4d9d3cf3a9db0944da"
server_key_iv: "8811d50dc6e6b3edb61e50b1fbafe57c"
server_key_version: "encryptionkey@0"
I20260924 10:43:39.278165 19718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "9168d0d358574e1eaf41c3136d7fdb62"
format_stamp: "Formatted at 2026-09-24 10:43:39 on dist-test-slave-j38h"
server_key: "4662ddb715b93d4d9d3cf3a9db0944da"
server_key_iv: "8811d50dc6e6b3edb61e50b1fbafe57c"
server_key_version: "encryptionkey@0"
I20260924 10:43:39.281941 19718 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:43:39.284332 19732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:39.285645 19718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:43:39.285799 19718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "9168d0d358574e1eaf41c3136d7fdb62"
format_stamp: "Formatted at 2026-09-24 10:43:39 on dist-test-slave-j38h"
server_key: "4662ddb715b93d4d9d3cf3a9db0944da"
server_key_iv: "8811d50dc6e6b3edb61e50b1fbafe57c"
server_key_version: "encryptionkey@0"
I20260924 10:43:39.285929 19718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:39.304222 19718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:39.305024 19718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:39.305294 19718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:39.305931 19718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:39.306928 19718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:39.307009 19718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:39.307080 19718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:39.307137 19718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:39.316524 19718 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:46173
I20260924 10:43:39.316547 19845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:46173 every 8 connection(s)
I20260924 10:43:39.317554 19718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:43:39.319705 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 19718
I20260924 10:43:39.319818 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:43:39.320132 2080 external_mini_cluster.cc:1428] Setting key 6c48f79d3f931767b716d983f1236ef0
I20260924 10:43:39.322115 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:37021
--builtin_ntp_servers=127.2.8.20:46663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:43:39.327973 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.319475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:52297 (local address 127.2.8.62:37021)
0924 10:43:39.319653 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:39.319659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:39.320919 (+ 1260us) server_negotiation.cc:407] Connection header received
0924 10:43:39.322400 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:39.322404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:39.322460 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:39.322546 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:39.323730 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.324315 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.325714 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.325975 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.326103 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.327410 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:39.327414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:39.327417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:39.327435 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:39.327462 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:39.327470 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:39.327561 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:39.327745 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 10:43:39.327818 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 10:43:39.328843 19848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.319980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:37021 (local address 127.2.8.1:52297)
0924 10:43:39.320780 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:39.320833 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:39.322137 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:39.322762 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:39.322769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:39.322857 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:39.323531 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:39.323552 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.324456 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.324459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:39.325566 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:39.325576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.326090 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.326148 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:39.326356 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.327122 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:43:39.327123 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:39.327165 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:43:39.327231 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:39.327576 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:39.327586 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:39.327592 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:43:39.327734 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 10:43:39.328016 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":158,"threads_started":1}
I20260924 10:43:39.330242 19846 heartbeater.cc:347] Connected to a master server at 127.2.8.62:37021
I20260924 10:43:39.330524 19846 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:39.331228 19846 heartbeater.cc:511] Master 127.2.8.62:37021 requested a full tablet report, sending...
I20260924 10:43:39.333086 19658 ts_manager.cc:194] Registered new tserver with Master: 9168d0d358574e1eaf41c3136d7fdb62 (127.2.8.1:46173)
I20260924 10:43:39.334628 19658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.8.1:52297
W20260924 10:43:39.437759 19849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:39.438074 19849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:39.438144 19849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:39.442268 19849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:39.442363 19849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:39.442453 19849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:43:39.447300 19849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:46663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:37021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:39.448441 19849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:39.449884 19849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:39.459635 19854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:39.459625 19855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:39.459759 19849 server_base.cc:1034] running on GCE node
W20260924 10:43:39.459708 19857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:39.460417 19849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:39.461177 19849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:39.462337 19849 hybrid_clock.cc:648] HybridClock initialized: now 1790246619462324 us; error 25 us; skew 500 ppm
I20260924 10:43:39.464143 19849 webserver.cc:460] Webserver started at http://127.2.8.2:38487/ using document root <none> and password file <none>
I20260924 10:43:39.464741 19849 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:39.464805 19849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:39.465090 19849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:39.467440 19849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "06f42e4a6a154d9d901c2e97a347b6f5"
format_stamp: "Formatted at 2026-09-24 10:43:39 on dist-test-slave-j38h"
server_key: "ade7249294152f0de6227a63f9492d93"
server_key_iv: "bf01460849adc3350b2eb886c93821ac"
server_key_version: "encryptionkey@0"
I20260924 10:43:39.468024 19849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "06f42e4a6a154d9d901c2e97a347b6f5"
format_stamp: "Formatted at 2026-09-24 10:43:39 on dist-test-slave-j38h"
server_key: "ade7249294152f0de6227a63f9492d93"
server_key_iv: "bf01460849adc3350b2eb886c93821ac"
server_key_version: "encryptionkey@0"
I20260924 10:43:39.471745 19849 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:43:39.474208 19863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:39.475603 19849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:43:39.475729 19849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "06f42e4a6a154d9d901c2e97a347b6f5"
format_stamp: "Formatted at 2026-09-24 10:43:39 on dist-test-slave-j38h"
server_key: "ade7249294152f0de6227a63f9492d93"
server_key_iv: "bf01460849adc3350b2eb886c93821ac"
server_key_version: "encryptionkey@0"
I20260924 10:43:39.475843 19849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:39.509994 19849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:39.510764 19849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:39.510987 19849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:39.511633 19849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:39.512605 19849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:39.512683 19849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:39.512750 19849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:39.512805 19849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:39.522315 19849 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:44233
I20260924 10:43:39.522336 19976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:44233 every 8 connection(s)
I20260924 10:43:39.523264 19849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:43:39.529251 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 19849
I20260924 10:43:39.529398 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:43:39.529680 2080 external_mini_cluster.cc:1428] Setting key 87cd0eb8be3f0527cc085049d36307b9
I20260924 10:43:39.531759 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:37021
--builtin_ntp_servers=127.2.8.20:46663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:43:39.533442 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.525264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:33733 (local address 127.2.8.62:37021)
0924 10:43:39.525448 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:39.525451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:39.526615 (+ 1164us) server_negotiation.cc:407] Connection header received
0924 10:43:39.528217 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:39.528221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:39.528280 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:39.528365 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:39.529309 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.529965 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.531173 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.531391 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.531471 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.532784 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:39.532787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:39.532789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:39.532801 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:39.532821 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:39.532829 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:39.532897 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:39.533190 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 10:43:39.533309 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:43:39.534662 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.525649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:37021 (local address 127.2.8.2:33733)
0924 10:43:39.526458 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:39.526517 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:39.527989 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:39.528553 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:39.528561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:39.528647 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:39.529152 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:39.529165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.530127 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.530131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:39.531014 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:39.531024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.531508 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.531569 (+ 61us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:39.531791 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.532435 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:43:39.532439 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:39.532499 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:43:39.532589 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:39.532926 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:39.532938 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:39.532955 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 10:43:39.533635 (+ 680us) client_negotiation.cc:240] Negotiation successful
0924 10:43:39.533913 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260924 10:43:39.535917 19977 heartbeater.cc:347] Connected to a master server at 127.2.8.62:37021
I20260924 10:43:39.536191 19977 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:39.536892 19977 heartbeater.cc:511] Master 127.2.8.62:37021 requested a full tablet report, sending...
I20260924 10:43:39.538120 19658 ts_manager.cc:194] Registered new tserver with Master: 06f42e4a6a154d9d901c2e97a347b6f5 (127.2.8.2:44233)
I20260924 10:43:39.538700 19658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.8.2:33733
W20260924 10:43:39.643605 19980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:39.643890 19980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:39.643930 19980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:39.648139 19980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:39.648195 19980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:39.648250 19980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:43:39.653013 19980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:46663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:37021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:39.654109 19980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:39.655493 19980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:39.665534 19986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:39.665666 19980 server_base.cc:1034] running on GCE node
W20260924 10:43:39.665576 19985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:39.665548 19988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:39.666225 19980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:39.666920 19980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:39.668082 19980 hybrid_clock.cc:648] HybridClock initialized: now 1790246619668065 us; error 29 us; skew 500 ppm
I20260924 10:43:39.669968 19980 webserver.cc:460] Webserver started at http://127.2.8.3:37257/ using document root <none> and password file <none>
I20260924 10:43:39.670562 19980 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:39.670624 19980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:39.670869 19980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:39.672937 19980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "85b69478dc934930af147bf3f46c8246"
format_stamp: "Formatted at 2026-09-24 10:43:39 on dist-test-slave-j38h"
server_key: "b41f5fc8558fbdb4b0220a52fb7bbeb0"
server_key_iv: "2166d5ab9298851047faef584cfe21b2"
server_key_version: "encryptionkey@0"
I20260924 10:43:39.673592 19980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "85b69478dc934930af147bf3f46c8246"
format_stamp: "Formatted at 2026-09-24 10:43:39 on dist-test-slave-j38h"
server_key: "b41f5fc8558fbdb4b0220a52fb7bbeb0"
server_key_iv: "2166d5ab9298851047faef584cfe21b2"
server_key_version: "encryptionkey@0"
I20260924 10:43:39.677340 19980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:43:39.680181 19994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:39.681512 19980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:43:39.681612 19980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "85b69478dc934930af147bf3f46c8246"
format_stamp: "Formatted at 2026-09-24 10:43:39 on dist-test-slave-j38h"
server_key: "b41f5fc8558fbdb4b0220a52fb7bbeb0"
server_key_iv: "2166d5ab9298851047faef584cfe21b2"
server_key_version: "encryptionkey@0"
I20260924 10:43:39.681753 19980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:39.705220 19980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:39.705953 19980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:39.706166 19980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:39.706805 19980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:39.707770 19980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:39.707816 19980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:39.707895 19980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:39.707934 19980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:39.717603 19980 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:37749
I20260924 10:43:39.717620 20107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:37749 every 8 connection(s)
I20260924 10:43:39.718528 19980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:43:39.718950 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 19980
I20260924 10:43:39.719062 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:43:39.719375 2080 external_mini_cluster.cc:1428] Setting key 9e3575e27fa5979e9a082078d151949a
I20260924 10:43:39.728379 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.720471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:36741 (local address 127.2.8.62:37021)
0924 10:43:39.720623 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:39.720627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:39.721964 (+ 1337us) server_negotiation.cc:407] Connection header received
0924 10:43:39.723456 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:39.723459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:39.723503 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:39.723575 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:39.724539 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.725175 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.726409 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.726664 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.726800 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.727859 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:39.727862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:39.727865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:39.727883 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:39.727901 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:39.727908 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:39.727997 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:39.728171 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 10:43:39.728234 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:43:39.729363 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.720922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:37021 (local address 127.2.8.3:36741)
0924 10:43:39.721797 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:39.721855 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:39.723265 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:39.723742 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:39.723749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:39.723834 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:39.724329 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:39.724341 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.725319 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.725323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:39.726278 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:39.726287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.726786 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.726841 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:39.727016 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.727595 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:43:39.727596 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:39.727637 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:43:39.727700 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:39.728010 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:39.728021 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:39.728027 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 10:43:39.728160 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 10:43:39.728457 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":140,"threads_started":1}
I20260924 10:43:39.730544 20108 heartbeater.cc:347] Connected to a master server at 127.2.8.62:37021
I20260924 10:43:39.730783 20108 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:39.731300 20108 heartbeater.cc:511] Master 127.2.8.62:37021 requested a full tablet report, sending...
I20260924 10:43:39.732327 19658 ts_manager.cc:194] Registered new tserver with Master: 85b69478dc934930af147bf3f46c8246 (127.2.8.3:37749)
I20260924 10:43:39.733019 19658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.8.3:36741
I20260924 10:43:39.734236 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:43:39.739679 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.736183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.2.8.62:37021)
0924 10:43:39.736312 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:39.736316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:39.736500 (+ 184us) server_negotiation.cc:407] Connection header received
0924 10:43:39.736592 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:39.736594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:39.736630 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:39.736694 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:39.736900 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.737479 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.738291 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.738505 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.738578 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.739355 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:39.739357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:39.739359 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:39.739369 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:39.739383 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:39.739389 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:39.739441 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:39.739536 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 10:43:39.739578 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:43:39.742653 19658 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:43:39.745687 19658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:43:39.760339 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.755078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.2.8.2:44233)
0924 10:43:39.755531 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:39.755546 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:39.755653 (+ 107us) server_negotiation.cc:407] Connection header received
0924 10:43:39.755753 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:39.755758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:39.755956 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:39.756129 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:39.756298 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.757097 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.759076 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.759597 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.759726 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.759926 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:39.760060 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 10:43:39.760160 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":138,"threads_started":1}
I20260924 10:43:39.760334 20122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.754574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:44233 (local address 127.0.0.1:32832)
0924 10:43:39.755528 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:39.755543 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:39.755645 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:39.756104 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:39.756106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:39.756124 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:39.756201 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:39.756207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.757209 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.757215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:39.758958 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:39.758965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.759735 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.759739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:39.759840 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.759875 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 10:43:39.760005 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 10:43:39.760153 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":167,"thread_start_us":90,"threads_started":1}
I20260924 10:43:39.760360 20119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.753857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:37749 (local address 127.0.0.1:38218)
0924 10:43:39.754240 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:39.754545 (+ 305us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:39.754662 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:39.755482 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:39.755485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:39.755516 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:39.755846 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:39.755852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.757337 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.757340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:39.758764 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:39.758775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.759768 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.759772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:39.759888 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.759909 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:43:39.759979 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 10:43:39.760153 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":360,"spinlock_wait_cycles":11904,"thread_start_us":86,"threads_started":1}
I20260924 10:43:39.761379 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.754252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:46173 (local address 127.0.0.1:56700)
0924 10:43:39.754466 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:39.754524 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:39.754662 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:39.755475 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:39.755481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:39.755515 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:39.755690 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:39.755697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.757372 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.757376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:39.758677 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:39.758686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.759768 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.759772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:39.761040 (+ 1268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.761070 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:43:39.761180 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 10:43:39.761251 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":699,"spinlock_wait_cycles":11008,"thread_start_us":81,"threads_started":1}
I20260924 10:43:39.761502 20123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.754651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.2.8.1:46173)
0924 10:43:39.754990 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:39.755000 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:39.755021 (+ 21us) server_negotiation.cc:407] Connection header received
0924 10:43:39.755084 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:39.755088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:39.755287 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:39.755833 (+ 546us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:39.756002 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.757220 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.758806 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.759622 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.759871 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.760108 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:39.761232 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0924 10:43:39.761343 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":140,"threads_started":1}
I20260924 10:43:39.762316 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.754042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.2.8.3:37749)
0924 10:43:39.754958 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:39.754970 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:39.754998 (+ 28us) server_negotiation.cc:407] Connection header received
0924 10:43:39.755076 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:39.755082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:39.755287 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:39.755620 (+ 333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:39.755974 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.757220 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.758865 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.759622 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.761909 (+ 2287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.762087 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:39.762126 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 10:43:39.762195 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":132,"threads_started":1}
I20260924 10:43:39.763557 19911 tablet_service.cc:1467] Processing CreateTablet for tablet c96de3f0c19f47ae9f1ad10bc2f2ed34 (DEFAULT_TABLE table=test-table [id=38c1a4ae397b4b69b37af79c6f1ee7e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:39.764259 19780 tablet_service.cc:1467] Processing CreateTablet for tablet c96de3f0c19f47ae9f1ad10bc2f2ed34 (DEFAULT_TABLE table=test-table [id=38c1a4ae397b4b69b37af79c6f1ee7e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:39.764724 19911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96de3f0c19f47ae9f1ad10bc2f2ed34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:39.765291 19780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96de3f0c19f47ae9f1ad10bc2f2ed34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:39.764887 20042 tablet_service.cc:1467] Processing CreateTablet for tablet c96de3f0c19f47ae9f1ad10bc2f2ed34 (DEFAULT_TABLE table=test-table [id=38c1a4ae397b4b69b37af79c6f1ee7e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:39.765909 20042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c96de3f0c19f47ae9f1ad10bc2f2ed34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:39.771612 20125 tablet_bootstrap.cc:492] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 9168d0d358574e1eaf41c3136d7fdb62: Bootstrap starting.
I20260924 10:43:39.773110 20127 tablet_bootstrap.cc:492] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246: Bootstrap starting.
I20260924 10:43:39.773255 20126 tablet_bootstrap.cc:492] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5: Bootstrap starting.
I20260924 10:43:39.774225 20125 tablet_bootstrap.cc:654] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 9168d0d358574e1eaf41c3136d7fdb62: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:39.775120 20125 log.cc:824] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 9168d0d358574e1eaf41c3136d7fdb62: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:39.775175 20126 tablet_bootstrap.cc:654] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:39.775239 20127 tablet_bootstrap.cc:654] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:39.775996 20126 log.cc:824] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:39.776026 20127 log.cc:824] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:39.777613 20125 tablet_bootstrap.cc:492] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 9168d0d358574e1eaf41c3136d7fdb62: No bootstrap required, opened a new log
I20260924 10:43:39.777824 20125 ts_tablet_manager.cc:1397] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 9168d0d358574e1eaf41c3136d7fdb62: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:43:39.777945 20126 tablet_bootstrap.cc:492] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5: No bootstrap required, opened a new log
I20260924 10:43:39.778059 20127 tablet_bootstrap.cc:492] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246: No bootstrap required, opened a new log
I20260924 10:43:39.778139 20126 ts_tablet_manager.cc:1397] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 10:43:39.778236 20127 ts_tablet_manager.cc:1397] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 10:43:39.781744 20127 raft_consensus.cc:348] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69478dc934930af147bf3f46c8246" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37749 } } peers { permanent_uuid: "06f42e4a6a154d9d901c2e97a347b6f5" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44233 } } peers { permanent_uuid: "9168d0d358574e1eaf41c3136d7fdb62" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 46173 } }
I20260924 10:43:39.781998 20127 raft_consensus.cc:374] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:39.782050 20127 raft_consensus.cc:729] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b69478dc934930af147bf3f46c8246, State: Initialized, Role: FOLLOWER
I20260924 10:43:39.782400 20126 raft_consensus.cc:348] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69478dc934930af147bf3f46c8246" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37749 } } peers { permanent_uuid: "06f42e4a6a154d9d901c2e97a347b6f5" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44233 } } peers { permanent_uuid: "9168d0d358574e1eaf41c3136d7fdb62" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 46173 } }
I20260924 10:43:39.782534 20127 consensus_queue.cc:260] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69478dc934930af147bf3f46c8246" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37749 } } peers { permanent_uuid: "06f42e4a6a154d9d901c2e97a347b6f5" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44233 } } peers { permanent_uuid: "9168d0d358574e1eaf41c3136d7fdb62" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 46173 } }
I20260924 10:43:39.782644 20126 raft_consensus.cc:374] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:39.782687 20126 raft_consensus.cc:729] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f42e4a6a154d9d901c2e97a347b6f5, State: Initialized, Role: FOLLOWER
I20260924 10:43:39.782621 20125 raft_consensus.cc:348] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 9168d0d358574e1eaf41c3136d7fdb62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69478dc934930af147bf3f46c8246" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37749 } } peers { permanent_uuid: "06f42e4a6a154d9d901c2e97a347b6f5" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44233 } } peers { permanent_uuid: "9168d0d358574e1eaf41c3136d7fdb62" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 46173 } }
I20260924 10:43:39.782841 20125 raft_consensus.cc:374] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 9168d0d358574e1eaf41c3136d7fdb62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:39.782900 20125 raft_consensus.cc:729] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 9168d0d358574e1eaf41c3136d7fdb62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9168d0d358574e1eaf41c3136d7fdb62, State: Initialized, Role: FOLLOWER
I20260924 10:43:39.783138 20126 consensus_queue.cc:260] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69478dc934930af147bf3f46c8246" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37749 } } peers { permanent_uuid: "06f42e4a6a154d9d901c2e97a347b6f5" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44233 } } peers { permanent_uuid: "9168d0d358574e1eaf41c3136d7fdb62" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 46173 } }
I20260924 10:43:39.783270 20125 consensus_queue.cc:260] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 9168d0d358574e1eaf41c3136d7fdb62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69478dc934930af147bf3f46c8246" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37749 } } peers { permanent_uuid: "06f42e4a6a154d9d901c2e97a347b6f5" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44233 } } peers { permanent_uuid: "9168d0d358574e1eaf41c3136d7fdb62" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 46173 } }
I20260924 10:43:39.783593 20127 ts_tablet_manager.cc:1428] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260924 10:43:39.783799 20108 heartbeater.cc:503] Master 127.2.8.62:37021 was elected leader, sending a full tablet report...
I20260924 10:43:39.784080 20126 ts_tablet_manager.cc:1428] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 10:43:39.784080 20125 ts_tablet_manager.cc:1428] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 9168d0d358574e1eaf41c3136d7fdb62: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 10:43:39.784436 19846 heartbeater.cc:503] Master 127.2.8.62:37021 was elected leader, sending a full tablet report...
I20260924 10:43:39.784457 19977 heartbeater.cc:503] Master 127.2.8.62:37021 was elected leader, sending a full tablet report...
W20260924 10:43:39.818866 19847 tablet.cc:2369] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 9168d0d358574e1eaf41c3136d7fdb62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:43:39.854202 20131 raft_consensus.cc:482] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:43:39.854485 20131 raft_consensus.cc:504] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69478dc934930af147bf3f46c8246" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37749 } } peers { permanent_uuid: "06f42e4a6a154d9d901c2e97a347b6f5" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44233 } } peers { permanent_uuid: "9168d0d358574e1eaf41c3136d7fdb62" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 46173 } }
I20260924 10:43:39.855679 20131 leader_election.cc:288] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06f42e4a6a154d9d901c2e97a347b6f5 (127.2.8.2:44233), 9168d0d358574e1eaf41c3136d7fdb62 (127.2.8.1:46173)
I20260924 10:43:39.860661 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.855995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:44049 (local address 127.2.8.2:44233)
0924 10:43:39.856135 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:39.856139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:39.856158 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:43:39.856293 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:39.856297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:39.856352 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:39.856430 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:39.856765 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.857591 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.859300 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.859898 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.860034 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.860098 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:39.860458 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 10:43:39.860527 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 10:43:39.860661 20123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.856266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:55043 (local address 127.2.8.1:46173)
0924 10:43:39.856435 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:39.856439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:39.856458 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:43:39.856620 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:39.856622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:39.856663 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:39.856764 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:39.857039 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.857737 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.859296 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.859897 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.860034 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.860098 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:39.860452 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 10:43:39.860527 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 10:43:39.860690 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.855902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:44233 (local address 127.2.8.3:44049)
0924 10:43:39.856052 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:39.856069 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:39.856164 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:39.856477 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:39.856479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:39.856501 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:39.856645 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:39.856652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.857736 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.857758 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:39.859168 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:39.859174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.860056 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.860060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:39.860284 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.860338 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 10:43:39.860419 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:43:39.860535 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":186,"spinlock_wait_cycles":4992}
I20260924 10:43:39.860702 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.855925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:46173 (local address 127.2.8.3:55043)
0924 10:43:39.856370 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:39.856381 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:39.856497 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:39.856796 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:39.856798 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:39.856816 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:39.856892 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:39.856896 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.857853 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.857856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:39.859162 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:39.859173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.860056 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.860060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:39.860258 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.860337 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 10:43:39.860419 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 10:43:39.860528 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":125,"thread_start_us":115,"threads_started":1}
I20260924 10:43:39.861408 19800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96de3f0c19f47ae9f1ad10bc2f2ed34" candidate_uuid: "85b69478dc934930af147bf3f46c8246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9168d0d358574e1eaf41c3136d7fdb62" is_pre_election: true
I20260924 10:43:39.861423 19931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96de3f0c19f47ae9f1ad10bc2f2ed34" candidate_uuid: "85b69478dc934930af147bf3f46c8246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f42e4a6a154d9d901c2e97a347b6f5" is_pre_election: true
I20260924 10:43:39.861754 19800 raft_consensus.cc:2454] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 9168d0d358574e1eaf41c3136d7fdb62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b69478dc934930af147bf3f46c8246 in term 0.
I20260924 10:43:39.861757 19931 raft_consensus.cc:2454] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b69478dc934930af147bf3f46c8246 in term 0.
I20260924 10:43:39.862289 19995 leader_election.cc:302] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85b69478dc934930af147bf3f46c8246, 9168d0d358574e1eaf41c3136d7fdb62; no voters:
I20260924 10:43:39.862601 20131 raft_consensus.cc:2781] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:43:39.862679 20131 raft_consensus.cc:482] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:43:39.862709 20131 raft_consensus.cc:3037] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:39.863741 20131 raft_consensus.cc:504] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69478dc934930af147bf3f46c8246" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37749 } } peers { permanent_uuid: "06f42e4a6a154d9d901c2e97a347b6f5" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44233 } } peers { permanent_uuid: "9168d0d358574e1eaf41c3136d7fdb62" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 46173 } }
I20260924 10:43:39.864156 20131 leader_election.cc:288] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [CANDIDATE]: Term 1 election: Requested vote from peers 06f42e4a6a154d9d901c2e97a347b6f5 (127.2.8.2:44233), 9168d0d358574e1eaf41c3136d7fdb62 (127.2.8.1:46173)
I20260924 10:43:39.864470 19931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96de3f0c19f47ae9f1ad10bc2f2ed34" candidate_uuid: "85b69478dc934930af147bf3f46c8246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f42e4a6a154d9d901c2e97a347b6f5"
I20260924 10:43:39.864624 19800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c96de3f0c19f47ae9f1ad10bc2f2ed34" candidate_uuid: "85b69478dc934930af147bf3f46c8246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9168d0d358574e1eaf41c3136d7fdb62"
I20260924 10:43:39.864595 19931 raft_consensus.cc:3037] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:39.864768 19800 raft_consensus.cc:3037] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 9168d0d358574e1eaf41c3136d7fdb62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:39.865748 19931 raft_consensus.cc:2454] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b69478dc934930af147bf3f46c8246 in term 1.
I20260924 10:43:39.865890 19800 raft_consensus.cc:2454] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 9168d0d358574e1eaf41c3136d7fdb62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b69478dc934930af147bf3f46c8246 in term 1.
I20260924 10:43:39.866135 19997 leader_election.cc:302] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f42e4a6a154d9d901c2e97a347b6f5, 85b69478dc934930af147bf3f46c8246; no voters:
I20260924 10:43:39.866331 20131 raft_consensus.cc:2781] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:39.866565 20131 raft_consensus.cc:686] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [term 1 LEADER]: Becoming Leader. State: Replica: 85b69478dc934930af147bf3f46c8246, State: Running, Role: LEADER
I20260924 10:43:39.866894 20131 consensus_queue.cc:237] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69478dc934930af147bf3f46c8246" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37749 } } peers { permanent_uuid: "06f42e4a6a154d9d901c2e97a347b6f5" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44233 } } peers { permanent_uuid: "9168d0d358574e1eaf41c3136d7fdb62" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 46173 } }
I20260924 10:43:39.870332 19657 catalog_manager.cc:5495] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85b69478dc934930af147bf3f46c8246 (127.2.8.3). New cstate: current_term: 1 leader_uuid: "85b69478dc934930af147bf3f46c8246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69478dc934930af147bf3f46c8246" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 37749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f42e4a6a154d9d901c2e97a347b6f5" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9168d0d358574e1eaf41c3136d7fdb62" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 46173 } health_report { overall_health: UNKNOWN } } }
I20260924 10:43:39.882910 20120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.879220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.2.8.3:37749)
0924 10:43:39.879364 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:39.879368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:39.879424 (+ 56us) server_negotiation.cc:407] Connection header received
0924 10:43:39.879510 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:39.879513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:39.879564 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:39.879689 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:43:39.879932 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.880466 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.881427 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:39.881644 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:39.881792 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:39.882119 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:43:39.882526 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:43:39.882673 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:39.882710 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 10:43:39.882759 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:43:39.888780 19931 raft_consensus.cc:1261] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5 [term 1 FOLLOWER]: Refusing update from remote peer 85b69478dc934930af147bf3f46c8246: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:43:39.888780 19800 raft_consensus.cc:1261] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 9168d0d358574e1eaf41c3136d7fdb62 [term 1 FOLLOWER]: Refusing update from remote peer 85b69478dc934930af147bf3f46c8246: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:43:39.889601 20131 consensus_queue.cc:1035] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f42e4a6a154d9d901c2e97a347b6f5" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:43:39.889779 20135 consensus_queue.cc:1035] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9168d0d358574e1eaf41c3136d7fdb62" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 46173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:43:39.898844 20140 mvcc.cc:205] Tried to move back new op lower bound from 7332850155056914432 to 7332850154978004992. Current Snapshot: MvccSnapshot[applied={T|T < 7332850155056914432}]
I20260924 10:43:39.907012 19656 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:43:39.907162 19656 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:43:39.909561 19656 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9c28d2ae9784d7f8c44fa7d156c1414: Sending DeleteTablet for 3 replicas of tablet c96de3f0c19f47ae9f1ad10bc2f2ed34
I20260924 10:43:39.910363 19780 tablet_service.cc:1514] Processing DeleteTablet for tablet c96de3f0c19f47ae9f1ad10bc2f2ed34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:39 UTC) from {username='slave'} at 127.0.0.1:56700
I20260924 10:43:39.910359 19911 tablet_service.cc:1514] Processing DeleteTablet for tablet c96de3f0c19f47ae9f1ad10bc2f2ed34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:39 UTC) from {username='slave'} at 127.0.0.1:32832
I20260924 10:43:39.910848 20042 tablet_service.cc:1514] Processing DeleteTablet for tablet c96de3f0c19f47ae9f1ad10bc2f2ed34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:39 UTC) from {username='slave'} at 127.0.0.1:38218
I20260924 10:43:39.911090 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 19718
I20260924 10:43:39.912642 20149 tablet_replica.cc:330] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246: stopping tablet replica
I20260924 10:43:39.913002 20149 raft_consensus.cc:2229] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:43:39.913399 20147 tablet_replica.cc:330] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5: stopping tablet replica
I20260924 10:43:39.913439 20149 raft_consensus.cc:2258] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:43:39.913719 20147 raft_consensus.cc:2229] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:43:39.914378 20147 raft_consensus.cc:2258] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:43:39.914849 20149 ts_tablet_manager.cc:1905] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:43:39.915848 20147 ts_tablet_manager.cc:1905] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:43:39.917655 20149 ts_tablet_manager.cc:1918] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:43:39.917760 20149 log.cc:1196] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/c96de3f0c19f47ae9f1ad10bc2f2ed34
I20260924 10:43:39.918073 20149 ts_tablet_manager.cc:1939] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 85b69478dc934930af147bf3f46c8246: Deleting consensus metadata
I20260924 10:43:39.918738 20147 ts_tablet_manager.cc:1918] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:43:39.918835 20147 log.cc:1196] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/c96de3f0c19f47ae9f1ad10bc2f2ed34
I20260924 10:43:39.919170 20147 ts_tablet_manager.cc:1939] T c96de3f0c19f47ae9f1ad10bc2f2ed34 P 06f42e4a6a154d9d901c2e97a347b6f5: Deleting consensus metadata
I20260924 10:43:39.919636 19644 catalog_manager.cc:4841] TS 85b69478dc934930af147bf3f46c8246 (127.2.8.3:37749): tablet c96de3f0c19f47ae9f1ad10bc2f2ed34 (table test-table [id=38c1a4ae397b4b69b37af79c6f1ee7e5]) successfully deleted
W20260924 10:43:39.920229 19642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.8.1:46173 (error 108)
I20260924 10:43:39.920495 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 19849
I20260924 10:43:39.920502 19644 catalog_manager.cc:4841] TS 06f42e4a6a154d9d901c2e97a347b6f5 (127.2.8.2:44233): tablet c96de3f0c19f47ae9f1ad10bc2f2ed34 (table test-table [id=38c1a4ae397b4b69b37af79c6f1ee7e5]) successfully deleted
I20260924 10:43:39.922191 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:39.921868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:46173 (local address 127.0.0.1:56712)
0924 10:43:39.922031 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:39.922105 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:46173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 10:43:39.922422 19642 catalog_manager.cc:4584] TS 9168d0d358574e1eaf41c3136d7fdb62 (127.2.8.1:46173): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c96de3f0c19f47ae9f1ad10bc2f2ed34: Network error: Client connection negotiation failed: client connection to 127.2.8.1:46173: connect: Connection refused (error 111)
I20260924 10:43:39.926488 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 19980
I20260924 10:43:39.932020 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 19627
2026-09-24T10:43:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1092 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:43:39 dist-test-slave-j38h krb5kdc[20153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:43:39 dist-test-slave-j38h krb5kdc[20153](info): set up 2 sockets
Sep 24 10:43:39 dist-test-slave-j38h krb5kdc[20153](info): commencing operation
krb5kdc: starting...
W20260924 10:43:40.988905 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:43:41 dist-test-slave-j38h krb5kdc[20153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:43:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:43:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:43:41.115865 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:40435
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:40435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:41.225931 20166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:41.226187 20166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:41.226233 20166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:41.230394 20166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:43:41.230453 20166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:41.230473 20166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:41.230517 20166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:43:41.230549 20166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:43:41.235743 20166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:40435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:40435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:41.236775 20166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:41.237816 20166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:41.246435 20174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:41.246580 20166 server_base.cc:1034] running on GCE node
W20260924 10:43:41.246582 20171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:41.246610 20172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:41.247184 20166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:41.248334 20166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:41.249522 20166 hybrid_clock.cc:648] HybridClock initialized: now 1790246621249492 us; error 42 us; skew 500 ppm
Sep 24 10:43:41 dist-test-slave-j38h krb5kdc[20153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:41.252254 20166 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:43:41.253293 20166 webserver.cc:460] Webserver started at http://127.2.8.62:34125/ using document root <none> and password file <none>
I20260924 10:43:41.253892 20166 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:41.253998 20166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:41.254232 20166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:41.256250 20166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "be6b187ea9d446159c6c6ca21635e484"
format_stamp: "Formatted at 2026-09-24 10:43:41 on dist-test-slave-j38h"
server_key: "e3509e5b025a3afacba2e259d3a7b263"
server_key_iv: "9f08c779671eea2e8fdcc59397d52265"
server_key_version: "encryptionkey@0"
I20260924 10:43:41.256815 20166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "be6b187ea9d446159c6c6ca21635e484"
format_stamp: "Formatted at 2026-09-24 10:43:41 on dist-test-slave-j38h"
server_key: "e3509e5b025a3afacba2e259d3a7b263"
server_key_iv: "9f08c779671eea2e8fdcc59397d52265"
server_key_version: "encryptionkey@0"
I20260924 10:43:41.260491 20166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:43:41.263005 20181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:41.264228 20166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:43:41.264369 20166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "be6b187ea9d446159c6c6ca21635e484"
format_stamp: "Formatted at 2026-09-24 10:43:41 on dist-test-slave-j38h"
server_key: "e3509e5b025a3afacba2e259d3a7b263"
server_key_iv: "9f08c779671eea2e8fdcc59397d52265"
server_key_version: "encryptionkey@0"
I20260924 10:43:41.264484 20166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:41.278218 20166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:41.281467 20166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:41.281682 20166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:41.289209 20166 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:40435
I20260924 10:43:41.289235 20233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:40435 every 8 connection(s)
I20260924 10:43:41.290210 20166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:43:41.292378 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 20166
I20260924 10:43:41.292524 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:43:41.292810 2080 external_mini_cluster.cc:1428] Setting key c97ab471287010d0e188c873f98d9849
I20260924 10:43:41.293224 20234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:41.300518 20234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484: Bootstrap starting.
Sep 24 10:43:41 dist-test-slave-j38h krb5kdc[20153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:43:41.303107 20234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:41.303843 20234 log.cc:824] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:41.305877 20234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484: No bootstrap required, opened a new log
I20260924 10:43:41.308044 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:41.294329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.2.8.62:40435)
0924 10:43:41.294771 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:41.294790 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:41.294833 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:43:41.295617 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:41.295641 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:41.296089 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:41.296464 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:41.297924 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:41.298928 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:41.300111 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:41.300675 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:41.300856 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:41.303621 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:41.303626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:41.303639 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:41.303666 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:41.305356 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:41.305814 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:41.305819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:41.305824 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:41.305928 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:41.306263 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:41.306266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:41.306267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:41.306629 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:41.306773 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:41.307045 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 10:43:41.307322 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":128,"threads_started":1}
I20260924 10:43:41.309723 20234 raft_consensus.cc:348] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6b187ea9d446159c6c6ca21635e484" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 40435 } }
I20260924 10:43:41.309964 20234 raft_consensus.cc:374] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:41.310055 20234 raft_consensus.cc:729] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6b187ea9d446159c6c6ca21635e484, State: Initialized, Role: FOLLOWER
I20260924 10:43:41.310485 20234 consensus_queue.cc:260] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6b187ea9d446159c6c6ca21635e484" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 40435 } }
I20260924 10:43:41.310634 20234 raft_consensus.cc:388] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:43:41.310721 20234 raft_consensus.cc:482] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:43:41.310837 20234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:41.311736 20234 raft_consensus.cc:504] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6b187ea9d446159c6c6ca21635e484" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 40435 } }
I20260924 10:43:41.312081 20234 leader_election.cc:302] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be6b187ea9d446159c6c6ca21635e484; no voters:
I20260924 10:43:41.312384 20234 leader_election.cc:288] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:43:41.312502 20239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:41.312747 20239 raft_consensus.cc:686] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [term 1 LEADER]: Becoming Leader. State: Replica: be6b187ea9d446159c6c6ca21635e484, State: Running, Role: LEADER
I20260924 10:43:41.313133 20239 consensus_queue.cc:237] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6b187ea9d446159c6c6ca21635e484" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 40435 } }
I20260924 10:43:41.313625 20234 sys_catalog.cc:564] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:43:41.314719 20240 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be6b187ea9d446159c6c6ca21635e484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6b187ea9d446159c6c6ca21635e484" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 40435 } } }
I20260924 10:43:41.315150 20240 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:41.315032 20241 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be6b187ea9d446159c6c6ca21635e484. Latest consensus state: current_term: 1 leader_uuid: "be6b187ea9d446159c6c6ca21635e484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be6b187ea9d446159c6c6ca21635e484" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 40435 } } }
I20260924 10:43:41.315483 20241 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484 [sys.catalog]: This master's current role is: LEADER
W20260924 10:43:41.318105 20254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:43:41.318195 20254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:43:41.318279 20251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:43:41.319002 20251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:43:41.324333 20251 catalog_manager.cc:1347] Generated new cluster ID: 4781ddb28d5f4e1eb85fa56b18c6f924
I20260924 10:43:41.324404 20251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:43:41.337909 20251 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:43:41.338848 20251 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:43:41.353283 20251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484: Generated new TSK 0
I20260924 10:43:41.354010 20251 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:43:41.393096 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:40435
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:41.505335 20260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:41.505607 20260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:41.505650 20260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:41.509677 20260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:41.509737 20260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:41.509796 20260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:43:41.515209 20260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:40435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:41.516384 20260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:41.517392 20260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:41.526880 20268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:41.526881 20266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:41.526909 20265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:41.526947 20260 server_base.cc:1034] running on GCE node
I20260924 10:43:41.527465 20260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:41.528162 20260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:41.529343 20260 hybrid_clock.cc:648] HybridClock initialized: now 1790246621529282 us; error 81 us; skew 500 ppm
Sep 24 10:43:41 dist-test-slave-j38h krb5kdc[20153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:41.539882 20260 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:43:41.540848 20260 webserver.cc:460] Webserver started at http://127.2.8.1:36931/ using document root <none> and password file <none>
I20260924 10:43:41.541465 20260 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:41.541530 20260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:41.541697 20260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:41.543691 20260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "92c12fa4af654548aa9ee7e5aa865570"
format_stamp: "Formatted at 2026-09-24 10:43:41 on dist-test-slave-j38h"
server_key: "52f7473ffeb32eb05ce3bbcf7a6683c4"
server_key_iv: "1c335ae2bcc84230197a40170ea6a107"
server_key_version: "encryptionkey@0"
I20260924 10:43:41.544219 20260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "92c12fa4af654548aa9ee7e5aa865570"
format_stamp: "Formatted at 2026-09-24 10:43:41 on dist-test-slave-j38h"
server_key: "52f7473ffeb32eb05ce3bbcf7a6683c4"
server_key_iv: "1c335ae2bcc84230197a40170ea6a107"
server_key_version: "encryptionkey@0"
I20260924 10:43:41.547943 20260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 10:43:41.550362 20275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:41.551553 20260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:43:41.551683 20260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "92c12fa4af654548aa9ee7e5aa865570"
format_stamp: "Formatted at 2026-09-24 10:43:41 on dist-test-slave-j38h"
server_key: "52f7473ffeb32eb05ce3bbcf7a6683c4"
server_key_iv: "1c335ae2bcc84230197a40170ea6a107"
server_key_version: "encryptionkey@0"
I20260924 10:43:41.551789 20260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:41.580683 20260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:41.584220 20260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:41.584436 20260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:41.585119 20260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:41.585969 20260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:41.586016 20260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:41.586089 20260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:41.586128 20260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:41.595042 20260 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:44701
I20260924 10:43:41.595126 20388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:44701 every 8 connection(s)
I20260924 10:43:41.595975 20260 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:43:41.599881 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 20260
I20260924 10:43:41.600047 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:43:41.600384 2080 external_mini_cluster.cc:1428] Setting key 78dd6d15d499049a76c991e5504ca9ee
Sep 24 10:43:41 dist-test-slave-j38h krb5kdc[20153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
I20260924 10:43:41.609568 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:41.597971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:41225 (local address 127.2.8.62:40435)
0924 10:43:41.598113 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:41.598117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:41.598926 (+ 809us) server_negotiation.cc:407] Connection header received
0924 10:43:41.599902 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:41.599905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:41.599976 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:41.600112 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:41.601459 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:41.602071 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:41.603205 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:41.603449 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:41.603560 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:41.605983 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:41.605993 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:41.605995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:41.606017 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:41.607456 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:41.607987 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:41.607992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:41.607998 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:41.608073 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:41.608423 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:41.608426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:41.608428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:41.608610 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:41.608718 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:41.609282 (+ 564us) server_negotiation.cc:299] Negotiation successful
0924 10:43:41.609408 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:43:41.610460 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:41.598340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:40435 (local address 127.2.8.1:41225)
0924 10:43:41.598773 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:41.598813 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:41.599653 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:41.600296 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:41.600303 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:41.600714 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:41.601309 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:41.601325 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:41.602199 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:41.602207 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:41.603079 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:41.603088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:41.603547 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:41.603601 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:41.603771 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:41.604380 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:43:41.604397 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:41.605816 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:41.607592 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:41.607598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:41.607612 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:41.607865 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:41.608165 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:41.608170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:41.608172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:41.608309 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:41.608718 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:41.608725 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:41.609040 (+ 315us) client_negotiation.cc:769] Sending connection context
0924 10:43:41.609244 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 10:43:41.609573 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":102,"threads_started":1}
I20260924 10:43:41.611639 20389 heartbeater.cc:347] Connected to a master server at 127.2.8.62:40435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
I20260924 10:43:41.611927 20389 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:41.612488 20389 heartbeater.cc:511] Master 127.2.8.62:40435 requested a full tablet report, sending...
I20260924 10:43:41.614154 20198 ts_manager.cc:194] Registered new tserver with Master: 92c12fa4af654548aa9ee7e5aa865570 (127.2.8.1:44701)
I20260924 10:43:41.615355 20198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:41225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:43:41.629513 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:40435
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:41.750387 20394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:41.750771 20394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:41.750876 20394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:41.758174 20394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:41.758268 20394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:41.758425 20394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:43:41.767274 20394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:40435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:41.768704 20394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:41.770076 20394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:41.780781 20399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:41.780789 20400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:41.781159 20394 server_base.cc:1034] running on GCE node
W20260924 10:43:41.781311 20402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:41.781730 20394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:41.782388 20394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:41.783581 20394 hybrid_clock.cc:648] HybridClock initialized: now 1790246621783546 us; error 45 us; skew 500 ppm
Sep 24 10:43:41 dist-test-slave-j38h krb5kdc[20153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:41.786450 20394 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:43:41.787352 20394 webserver.cc:460] Webserver started at http://127.2.8.2:35915/ using document root <none> and password file <none>
I20260924 10:43:41.787926 20394 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:41.788025 20394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:41.788277 20394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:41.790380 20394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "31f1c83cf26c46f9894f1a5534d717fa"
format_stamp: "Formatted at 2026-09-24 10:43:41 on dist-test-slave-j38h"
server_key: "cc7bbe01a8b0bc56becb591e6609ea74"
server_key_iv: "c8d0e8c96ec509edf3c6d6d46ff60b28"
server_key_version: "encryptionkey@0"
I20260924 10:43:41.790926 20394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "31f1c83cf26c46f9894f1a5534d717fa"
format_stamp: "Formatted at 2026-09-24 10:43:41 on dist-test-slave-j38h"
server_key: "cc7bbe01a8b0bc56becb591e6609ea74"
server_key_iv: "c8d0e8c96ec509edf3c6d6d46ff60b28"
server_key_version: "encryptionkey@0"
I20260924 10:43:41.794642 20394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:43:41.797128 20409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:41.798458 20394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:43:41.798599 20394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "31f1c83cf26c46f9894f1a5534d717fa"
format_stamp: "Formatted at 2026-09-24 10:43:41 on dist-test-slave-j38h"
server_key: "cc7bbe01a8b0bc56becb591e6609ea74"
server_key_iv: "c8d0e8c96ec509edf3c6d6d46ff60b28"
server_key_version: "encryptionkey@0"
I20260924 10:43:41.798718 20394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:41.816260 20394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:41.819415 20394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:41.819577 20394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:41.820119 20394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:41.820883 20394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:41.820930 20394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:41.821026 20394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:41.821069 20394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:41.830454 20394 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:35695
I20260924 10:43:41.830477 20522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:35695 every 8 connection(s)
I20260924 10:43:41.831382 20394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:43:41.836318 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 20394
I20260924 10:43:41.836495 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:43:41.836766 2080 external_mini_cluster.cc:1428] Setting key e651942b829a967c94e173344c23c05e
Sep 24 10:43:41 dist-test-slave-j38h krb5kdc[20153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:43:41.845642 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:41.833296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:43325 (local address 127.2.8.62:40435)
0924 10:43:41.833438 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:41.833447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:41.834281 (+ 834us) server_negotiation.cc:407] Connection header received
0924 10:43:41.835584 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:41.835587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:41.835650 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:41.835727 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:41.837080 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:41.837739 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:41.838908 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:41.839147 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:41.839258 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:41.841835 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:41.841839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:41.841842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:41.841873 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:41.843506 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:41.844029 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:41.844032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:41.844034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:41.844121 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:41.844471 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:41.844476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:41.844478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:41.844667 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:41.844769 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:41.845346 (+ 577us) server_negotiation.cc:299] Negotiation successful
0924 10:43:41.845475 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
I20260924 10:43:41.846568 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:41.833676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:40435 (local address 127.2.8.2:43325)
0924 10:43:41.834121 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:41.834164 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:41.835345 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:41.835914 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:41.835921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:41.836287 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:41.836902 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:41.836924 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:41.837867 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:41.837870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:41.838781 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:41.838790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:41.839244 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:41.839301 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:41.839488 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:41.840120 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:43:41.840136 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:41.841661 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:41.843624 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:41.843631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:41.843647 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:41.843919 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:41.844219 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:41.844222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:41.844224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:41.844366 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:41.844781 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:41.844790 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:41.845106 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 10:43:41.845304 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 10:43:41.845658 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":121,"threads_started":1}
I20260924 10:43:41.847779 20523 heartbeater.cc:347] Connected to a master server at 127.2.8.62:40435
I20260924 10:43:41.848032 20523 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:41.848570 20523 heartbeater.cc:511] Master 127.2.8.62:40435 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
I20260924 10:43:41.849745 20198 ts_manager.cc:194] Registered new tserver with Master: 31f1c83cf26c46f9894f1a5534d717fa (127.2.8.2:35695)
I20260924 10:43:41.850327 20198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:43325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:43:41.868014 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:40435
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:41.979640 20528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:41.979936 20528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:41.980028 20528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:41.984041 20528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:41.984141 20528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:41.984237 20528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:43:41.989612 20528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:40435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:41.990693 20528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:41.991725 20528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:42.001318 20533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:42.001312 20534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:42.001313 20536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:42.001711 20528 server_base.cc:1034] running on GCE node
I20260924 10:43:42.002173 20528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:42.002839 20528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:42.004006 20528 hybrid_clock.cc:648] HybridClock initialized: now 1790246622003988 us; error 28 us; skew 500 ppm
Sep 24 10:43:42 dist-test-slave-j38h krb5kdc[20153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246622, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:42.006920 20528 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:43:42.007899 20528 webserver.cc:460] Webserver started at http://127.2.8.3:33707/ using document root <none> and password file <none>
I20260924 10:43:42.008467 20528 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:42.008533 20528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:42.008805 20528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:42.010943 20528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "7e5dfe03173e4cada692b83e72f82202"
format_stamp: "Formatted at 2026-09-24 10:43:42 on dist-test-slave-j38h"
server_key: "b76ea2e5fbeee2c0dbc5952e8dafd0c0"
server_key_iv: "b9980ffff399d10f0154cb765196a399"
server_key_version: "encryptionkey@0"
I20260924 10:43:42.011543 20528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "7e5dfe03173e4cada692b83e72f82202"
format_stamp: "Formatted at 2026-09-24 10:43:42 on dist-test-slave-j38h"
server_key: "b76ea2e5fbeee2c0dbc5952e8dafd0c0"
server_key_iv: "b9980ffff399d10f0154cb765196a399"
server_key_version: "encryptionkey@0"
I20260924 10:43:42.015607 20528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:43:42.018132 20543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:42.019392 20528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:43:42.019619 20528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "7e5dfe03173e4cada692b83e72f82202"
format_stamp: "Formatted at 2026-09-24 10:43:42 on dist-test-slave-j38h"
server_key: "b76ea2e5fbeee2c0dbc5952e8dafd0c0"
server_key_iv: "b9980ffff399d10f0154cb765196a399"
server_key_version: "encryptionkey@0"
I20260924 10:43:42.019729 20528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:42.037878 20528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:42.041440 20528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:42.041664 20528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:42.042303 20528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:42.043174 20528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:42.043226 20528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:42.043308 20528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:42.043352 20528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:42.053234 20528 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:41701
I20260924 10:43:42.053251 20656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:41701 every 8 connection(s)
I20260924 10:43:42.054198 20528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:43:42.054666 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 20528
I20260924 10:43:42.054826 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:43:42.055207 2080 external_mini_cluster.cc:1428] Setting key 9d4488cfd1c4c8eaf1efbf04a785faea
Sep 24 10:43:42 dist-test-slave-j38h krb5kdc[20153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246622, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:43:42.068771 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.056235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:42293 (local address 127.2.8.62:40435)
0924 10:43:42.056378 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:42.056382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:42.057550 (+ 1168us) server_negotiation.cc:407] Connection header received
0924 10:43:42.058980 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:42.058983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:42.059056 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:42.059137 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:42.060306 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.060830 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.062153 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.062364 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.062450 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.065048 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:42.065052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:42.065054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:42.065086 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:42.066441 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:42.066945 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:42.066949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:42.066950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:42.067022 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:42.067460 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:42.067465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:42.067469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:42.067651 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:42.067735 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:42.068434 (+ 699us) server_negotiation.cc:299] Negotiation successful
0924 10:43:42.068593 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:43:42.069932 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.056809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:40435 (local address 127.2.8.3:42293)
0924 10:43:42.057394 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:42.057437 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:42.058735 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:42.059305 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:42.059312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:42.059673 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:42.060181 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:42.060196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.060947 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.060950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:42.062044 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:42.062055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.062466 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.062513 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:42.062667 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.063332 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:43:42.063349 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:42.064846 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:42.066568 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:42.066573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:42.066586 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:42.066852 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:42.067124 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:42.067127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:42.067129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:42.067268 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:42.067814 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:42.067820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:42.068152 (+ 332us) client_negotiation.cc:769] Sending connection context
0924 10:43:42.068398 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 10:43:42.068777 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":148,"threads_started":1}
I20260924 10:43:42.071070 20657 heartbeater.cc:347] Connected to a master server at 127.2.8.62:40435
I20260924 10:43:42.071331 20657 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:42.071851 20657 heartbeater.cc:511] Master 127.2.8.62:40435 requested a full tablet report, sending...
I20260924 10:43:42.072822 20198 ts_manager.cc:194] Registered new tserver with Master: 7e5dfe03173e4cada692b83e72f82202 (127.2.8.3:41701)
I20260924 10:43:42.073407 20198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:42293
I20260924 10:43:42.081522 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:43:42.091663 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.083600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41910 (local address 127.2.8.62:40435)
0924 10:43:42.083721 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:42.083732 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:42.084015 (+ 283us) server_negotiation.cc:407] Connection header received
0924 10:43:42.084164 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:42.084168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:42.084214 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:42.084307 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:42.085042 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.085564 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.086411 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.086663 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.086774 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.088358 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:42.088362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:42.088365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:42.088391 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:42.089803 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:42.090315 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:42.090319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:42.090321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:42.090400 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:42.090724 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:42.090727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:42.090729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:42.090939 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:42.091098 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:42.091363 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 10:43:42.091469 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:43:42.094748 20198 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:43:42.097760 20198 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:43:42.111991 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.106062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:44701 (local address 127.0.0.1:37912)
0924 10:43:42.106886 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:42.106938 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:42.107079 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:42.107829 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:42.107837 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:42.107866 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:42.108027 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:42.108034 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.109299 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.109303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:42.110512 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:42.110521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.111458 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.111462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:42.111565 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.111604 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 10:43:42.111723 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 10:43:42.111841 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260924 10:43:42.112017 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.106245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37912 (local address 127.2.8.1:44701)
0924 10:43:42.106837 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:42.107277 (+ 440us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:42.107319 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:43:42.107404 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:42.107410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:42.107651 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:42.107813 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:42.108154 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.109176 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.110634 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.111328 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.111438 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.111584 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:42.111763 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 10:43:42.111860 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":158,"threads_started":1}
I20260924 10:43:42.112365 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.106330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:35695 (local address 127.0.0.1:39184)
0924 10:43:42.107541 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:42.107551 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:42.107632 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:42.108082 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:42.108085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:42.108111 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:42.108237 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:42.108245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.109542 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.109545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:42.110746 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:42.110755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.111972 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.111975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:42.112076 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.112104 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 10:43:42.112175 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 10:43:42.112248 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"thread_start_us":112,"threads_started":1}
I20260924 10:43:42.112771 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.106989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.2.8.2:35695)
0924 10:43:42.107424 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:42.107433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:42.107649 (+ 216us) server_negotiation.cc:407] Connection header received
0924 10:43:42.107741 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:42.107748 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:42.107938 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:42.108102 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:42.108369 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.109396 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.110919 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.111687 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.111778 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.111915 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:42.112520 (+ 605us) server_negotiation.cc:299] Negotiation successful
0924 10:43:42.112608 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":113,"threads_started":1}
I20260924 10:43:42.113958 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.106445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:41701 (local address 127.0.0.1:40376)
0924 10:43:42.107793 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:42.107805 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:42.107876 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:42.108541 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:42.108544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:42.108561 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:42.108656 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:42.108662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.110010 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.110013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:42.111137 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:42.111146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.112781 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.112784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:42.113660 (+ 876us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.113680 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:43:42.113773 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 10:43:42.113842 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1272,"thread_start_us":89,"threads_started":1}
I20260924 10:43:42.114305 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.106950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.2.8.3:41701)
0924 10:43:42.107498 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:42.107509 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:42.107963 (+ 454us) server_negotiation.cc:407] Connection header received
0924 10:43:42.108043 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:42.108048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:42.108240 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:42.108379 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:42.108778 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.109860 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.111258 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.112035 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.112568 (+ 533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.112719 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:42.113804 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0924 10:43:42.113898 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":160,"threads_started":1}
I20260924 10:43:42.115263 20323 tablet_service.cc:1467] Processing CreateTablet for tablet d218251c265848c7933942c5cc28bbf6 (DEFAULT_TABLE table=test-table [id=0000b2e0f05c4981a08ea717f386cb1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:42.115365 20457 tablet_service.cc:1467] Processing CreateTablet for tablet d218251c265848c7933942c5cc28bbf6 (DEFAULT_TABLE table=test-table [id=0000b2e0f05c4981a08ea717f386cb1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:42.116381 20323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d218251c265848c7933942c5cc28bbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:42.116382 20457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d218251c265848c7933942c5cc28bbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:42.116750 20591 tablet_service.cc:1467] Processing CreateTablet for tablet d218251c265848c7933942c5cc28bbf6 (DEFAULT_TABLE table=test-table [id=0000b2e0f05c4981a08ea717f386cb1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:42.117771 20591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d218251c265848c7933942c5cc28bbf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:42.123615 20674 tablet_bootstrap.cc:492] T d218251c265848c7933942c5cc28bbf6 P 92c12fa4af654548aa9ee7e5aa865570: Bootstrap starting.
I20260924 10:43:42.124274 20676 tablet_bootstrap.cc:492] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202: Bootstrap starting.
I20260924 10:43:42.124460 20675 tablet_bootstrap.cc:492] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa: Bootstrap starting.
I20260924 10:43:42.126754 20675 tablet_bootstrap.cc:654] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:42.126765 20674 tablet_bootstrap.cc:654] T d218251c265848c7933942c5cc28bbf6 P 92c12fa4af654548aa9ee7e5aa865570: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:42.126870 20676 tablet_bootstrap.cc:654] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:42.127686 20674 log.cc:824] T d218251c265848c7933942c5cc28bbf6 P 92c12fa4af654548aa9ee7e5aa865570: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:42.128072 20676 log.cc:824] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:42.128072 20675 log.cc:824] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:42.130033 20674 tablet_bootstrap.cc:492] T d218251c265848c7933942c5cc28bbf6 P 92c12fa4af654548aa9ee7e5aa865570: No bootstrap required, opened a new log
I20260924 10:43:42.130089 20675 tablet_bootstrap.cc:492] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa: No bootstrap required, opened a new log
I20260924 10:43:42.130089 20676 tablet_bootstrap.cc:492] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202: No bootstrap required, opened a new log
I20260924 10:43:42.130254 20674 ts_tablet_manager.cc:1397] T d218251c265848c7933942c5cc28bbf6 P 92c12fa4af654548aa9ee7e5aa865570: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 10:43:42.130290 20675 ts_tablet_manager.cc:1397] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:43:42.130290 20676 ts_tablet_manager.cc:1397] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:43:42.133719 20674 raft_consensus.cc:348] T d218251c265848c7933942c5cc28bbf6 P 92c12fa4af654548aa9ee7e5aa865570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c12fa4af654548aa9ee7e5aa865570" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44701 } } peers { permanent_uuid: "31f1c83cf26c46f9894f1a5534d717fa" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35695 } } peers { permanent_uuid: "7e5dfe03173e4cada692b83e72f82202" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 41701 } }
I20260924 10:43:42.133972 20674 raft_consensus.cc:374] T d218251c265848c7933942c5cc28bbf6 P 92c12fa4af654548aa9ee7e5aa865570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:42.134033 20674 raft_consensus.cc:729] T d218251c265848c7933942c5cc28bbf6 P 92c12fa4af654548aa9ee7e5aa865570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c12fa4af654548aa9ee7e5aa865570, State: Initialized, Role: FOLLOWER
I20260924 10:43:42.134291 20676 raft_consensus.cc:348] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c12fa4af654548aa9ee7e5aa865570" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44701 } } peers { permanent_uuid: "31f1c83cf26c46f9894f1a5534d717fa" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35695 } } peers { permanent_uuid: "7e5dfe03173e4cada692b83e72f82202" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 41701 } }
I20260924 10:43:42.134284 20675 raft_consensus.cc:348] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c12fa4af654548aa9ee7e5aa865570" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44701 } } peers { permanent_uuid: "31f1c83cf26c46f9894f1a5534d717fa" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35695 } } peers { permanent_uuid: "7e5dfe03173e4cada692b83e72f82202" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 41701 } }
I20260924 10:43:42.134560 20676 raft_consensus.cc:374] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:42.134560 20675 raft_consensus.cc:374] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:42.134510 20674 consensus_queue.cc:260] T d218251c265848c7933942c5cc28bbf6 P 92c12fa4af654548aa9ee7e5aa865570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c12fa4af654548aa9ee7e5aa865570" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44701 } } peers { permanent_uuid: "31f1c83cf26c46f9894f1a5534d717fa" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35695 } } peers { permanent_uuid: "7e5dfe03173e4cada692b83e72f82202" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 41701 } }
I20260924 10:43:42.134624 20676 raft_consensus.cc:729] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5dfe03173e4cada692b83e72f82202, State: Initialized, Role: FOLLOWER
I20260924 10:43:42.134624 20675 raft_consensus.cc:729] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f1c83cf26c46f9894f1a5534d717fa, State: Initialized, Role: FOLLOWER
I20260924 10:43:42.135054 20676 consensus_queue.cc:260] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c12fa4af654548aa9ee7e5aa865570" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44701 } } peers { permanent_uuid: "31f1c83cf26c46f9894f1a5534d717fa" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35695 } } peers { permanent_uuid: "7e5dfe03173e4cada692b83e72f82202" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 41701 } }
I20260924 10:43:42.135072 20675 consensus_queue.cc:260] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c12fa4af654548aa9ee7e5aa865570" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44701 } } peers { permanent_uuid: "31f1c83cf26c46f9894f1a5534d717fa" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35695 } } peers { permanent_uuid: "7e5dfe03173e4cada692b83e72f82202" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 41701 } }
I20260924 10:43:42.135586 20674 ts_tablet_manager.cc:1428] T d218251c265848c7933942c5cc28bbf6 P 92c12fa4af654548aa9ee7e5aa865570: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:43:42.136044 20676 ts_tablet_manager.cc:1428] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:43:42.136044 20389 heartbeater.cc:503] Master 127.2.8.62:40435 was elected leader, sending a full tablet report...
I20260924 10:43:42.136147 20657 heartbeater.cc:503] Master 127.2.8.62:40435 was elected leader, sending a full tablet report...
I20260924 10:43:42.136505 20523 heartbeater.cc:503] Master 127.2.8.62:40435 was elected leader, sending a full tablet report...
I20260924 10:43:42.136763 20675 ts_tablet_manager.cc:1428] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 10:43:42.251713 20681 raft_consensus.cc:482] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:43:42.251960 20681 raft_consensus.cc:504] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c12fa4af654548aa9ee7e5aa865570" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44701 } } peers { permanent_uuid: "31f1c83cf26c46f9894f1a5534d717fa" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35695 } } peers { permanent_uuid: "7e5dfe03173e4cada692b83e72f82202" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 41701 } }
I20260924 10:43:42.253382 20681 leader_election.cc:288] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92c12fa4af654548aa9ee7e5aa865570 (127.2.8.1:44701), 31f1c83cf26c46f9894f1a5534d717fa (127.2.8.2:35695)
I20260924 10:43:42.258181 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.254176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:48907 (local address 127.2.8.1:44701)
0924 10:43:42.254311 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:42.254316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:42.254355 (+ 39us) server_negotiation.cc:407] Connection header received
0924 10:43:42.254496 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:42.254499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:42.254553 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:42.254628 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:42.254945 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.255711 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.257201 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.257628 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.257754 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.257819 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:42.257979 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 10:43:42.258053 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 10:43:42.258210 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.253710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:44701 (local address 127.2.8.3:48907)
0924 10:43:42.254275 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:42.254297 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:42.254393 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:42.254657 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:42.254659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:42.254690 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:42.254787 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:42.254793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.255857 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.255861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:42.257077 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:42.257086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.257750 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.257753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:42.257847 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.257870 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:43:42.257964 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 10:43:42.258038 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":136,"spinlock_wait_cycles":1280,"thread_start_us":134,"threads_started":1}
I20260924 10:43:42.258219 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.254194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:59595 (local address 127.2.8.2:35695)
0924 10:43:42.254328 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:42.254334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:42.254352 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:43:42.254415 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:42.254417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:42.254494 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:42.254583 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:42.254900 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.255698 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.257231 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.257657 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.257774 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.257839 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:42.258024 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 10:43:42.258098 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 10:43:42.258220 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.253705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:35695 (local address 127.2.8.3:59595)
0924 10:43:42.253863 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:42.253880 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:42.253987 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:42.254589 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:42.254591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:42.254632 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:42.254787 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:42.254793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.255822 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.255825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:42.257110 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:42.257120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.257774 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.257778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:42.257890 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.257910 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:43:42.258009 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 10:43:42.258077 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":64,"spinlock_wait_cycles":24704}
I20260924 10:43:42.258832 20477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d218251c265848c7933942c5cc28bbf6" candidate_uuid: "7e5dfe03173e4cada692b83e72f82202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f1c83cf26c46f9894f1a5534d717fa" is_pre_election: true
I20260924 10:43:42.258852 20343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d218251c265848c7933942c5cc28bbf6" candidate_uuid: "7e5dfe03173e4cada692b83e72f82202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c12fa4af654548aa9ee7e5aa865570" is_pre_election: true
I20260924 10:43:42.259145 20343 raft_consensus.cc:2454] T d218251c265848c7933942c5cc28bbf6 P 92c12fa4af654548aa9ee7e5aa865570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e5dfe03173e4cada692b83e72f82202 in term 0.
I20260924 10:43:42.259152 20477 raft_consensus.cc:2454] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e5dfe03173e4cada692b83e72f82202 in term 0.
I20260924 10:43:42.259627 20547 leader_election.cc:302] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e5dfe03173e4cada692b83e72f82202, 92c12fa4af654548aa9ee7e5aa865570; no voters:
I20260924 10:43:42.259899 20681 raft_consensus.cc:2781] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:43:42.260000 20681 raft_consensus.cc:482] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:43:42.260061 20681 raft_consensus.cc:3037] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:42.261135 20681 raft_consensus.cc:504] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c12fa4af654548aa9ee7e5aa865570" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44701 } } peers { permanent_uuid: "31f1c83cf26c46f9894f1a5534d717fa" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35695 } } peers { permanent_uuid: "7e5dfe03173e4cada692b83e72f82202" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 41701 } }
I20260924 10:43:42.261466 20681 leader_election.cc:288] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [CANDIDATE]: Term 1 election: Requested vote from peers 92c12fa4af654548aa9ee7e5aa865570 (127.2.8.1:44701), 31f1c83cf26c46f9894f1a5534d717fa (127.2.8.2:35695)
I20260924 10:43:42.261822 20477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d218251c265848c7933942c5cc28bbf6" candidate_uuid: "7e5dfe03173e4cada692b83e72f82202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f1c83cf26c46f9894f1a5534d717fa"
I20260924 10:43:42.261828 20343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d218251c265848c7933942c5cc28bbf6" candidate_uuid: "7e5dfe03173e4cada692b83e72f82202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c12fa4af654548aa9ee7e5aa865570"
I20260924 10:43:42.261926 20477 raft_consensus.cc:3037] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:42.261974 20343 raft_consensus.cc:3037] T d218251c265848c7933942c5cc28bbf6 P 92c12fa4af654548aa9ee7e5aa865570 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:42.263023 20477 raft_consensus.cc:2454] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5dfe03173e4cada692b83e72f82202 in term 1.
I20260924 10:43:42.263106 20343 raft_consensus.cc:2454] T d218251c265848c7933942c5cc28bbf6 P 92c12fa4af654548aa9ee7e5aa865570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5dfe03173e4cada692b83e72f82202 in term 1.
I20260924 10:43:42.263316 20545 leader_election.cc:302] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31f1c83cf26c46f9894f1a5534d717fa, 7e5dfe03173e4cada692b83e72f82202; no voters:
I20260924 10:43:42.263499 20681 raft_consensus.cc:2781] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:42.263707 20681 raft_consensus.cc:686] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [term 1 LEADER]: Becoming Leader. State: Replica: 7e5dfe03173e4cada692b83e72f82202, State: Running, Role: LEADER
I20260924 10:43:42.264032 20681 consensus_queue.cc:237] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c12fa4af654548aa9ee7e5aa865570" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44701 } } peers { permanent_uuid: "31f1c83cf26c46f9894f1a5534d717fa" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35695 } } peers { permanent_uuid: "7e5dfe03173e4cada692b83e72f82202" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 41701 } }
I20260924 10:43:42.267128 20196 catalog_manager.cc:5495] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e5dfe03173e4cada692b83e72f82202 (127.2.8.3). New cstate: current_term: 1 leader_uuid: "7e5dfe03173e4cada692b83e72f82202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c12fa4af654548aa9ee7e5aa865570" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31f1c83cf26c46f9894f1a5534d717fa" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e5dfe03173e4cada692b83e72f82202" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 41701 } health_report { overall_health: HEALTHY } } }
I20260924 10:43:42.302070 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.298374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.2.8.3:41701)
0924 10:43:42.298529 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:42.298534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:42.298549 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:43:42.298661 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:42.298664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:42.298719 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:42.298831 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:43:42.299094 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.299602 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.300498 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.300717 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.300829 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.301181 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:43:42.301633 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:43:42.301790 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:42.301889 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 10:43:42.301932 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 10:43:42.305321 20658 tablet.cc:2369] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:43:42.308282 20343 raft_consensus.cc:1261] T d218251c265848c7933942c5cc28bbf6 P 92c12fa4af654548aa9ee7e5aa865570 [term 1 FOLLOWER]: Refusing update from remote peer 7e5dfe03173e4cada692b83e72f82202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:43:42.308283 20477 raft_consensus.cc:1261] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa [term 1 FOLLOWER]: Refusing update from remote peer 7e5dfe03173e4cada692b83e72f82202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:43:42.309109 20681 consensus_queue.cc:1035] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92c12fa4af654548aa9ee7e5aa865570" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:43:42.309285 20684 consensus_queue.cc:1035] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f1c83cf26c46f9894f1a5534d717fa" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 35695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:43:42.317480 20688 mvcc.cc:205] Tried to move back new op lower bound from 7332850164966940672 to 7332850164796063744. Current Snapshot: MvccSnapshot[applied={T|T < 7332850164966940672}]
I20260924 10:43:42.328457 20198 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:43:42.328619 20198 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:43:42.331157 20198 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be6b187ea9d446159c6c6ca21635e484: Sending DeleteTablet for 3 replicas of tablet d218251c265848c7933942c5cc28bbf6
I20260924 10:43:42.332020 20457 tablet_service.cc:1514] Processing DeleteTablet for tablet d218251c265848c7933942c5cc28bbf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:42 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:39184
I20260924 10:43:42.332017 20323 tablet_service.cc:1514] Processing DeleteTablet for tablet d218251c265848c7933942c5cc28bbf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:42 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:37912
I20260924 10:43:42.332098 20591 tablet_service.cc:1514] Processing DeleteTablet for tablet d218251c265848c7933942c5cc28bbf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:42 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:40376
I20260924 10:43:42.332563 20698 tablet_replica.cc:330] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202: stopping tablet replica
I20260924 10:43:42.332571 20697 tablet_replica.cc:330] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa: stopping tablet replica
I20260924 10:43:42.332871 20698 raft_consensus.cc:2229] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:43:42.333202 20697 raft_consensus.cc:2229] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:43:42.333302 20699 tablet_replica.cc:330] T d218251c265848c7933942c5cc28bbf6 P 92c12fa4af654548aa9ee7e5aa865570: stopping tablet replica
I20260924 10:43:42.333560 20699 raft_consensus.cc:2229] T d218251c265848c7933942c5cc28bbf6 P 92c12fa4af654548aa9ee7e5aa865570 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:43:42.333685 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 20260
I20260924 10:43:42.335789 20697 raft_consensus.cc:2258] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:43:42.337216 20697 ts_tablet_manager.cc:1905] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:43:42.337261 20698 raft_consensus.cc:2258] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:43:42.338837 20698 ts_tablet_manager.cc:1905] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:43:42.340106 20697 ts_tablet_manager.cc:1918] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:43:42.340214 20697 log.cc:1196] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/d218251c265848c7933942c5cc28bbf6
I20260924 10:43:42.340552 20697 ts_tablet_manager.cc:1939] T d218251c265848c7933942c5cc28bbf6 P 31f1c83cf26c46f9894f1a5534d717fa: Deleting consensus metadata
W20260924 10:43:42.341699 20185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.8.1:44701 (error 108)
I20260924 10:43:42.341890 20183 catalog_manager.cc:4841] TS 31f1c83cf26c46f9894f1a5534d717fa (127.2.8.2:35695): tablet d218251c265848c7933942c5cc28bbf6 (table test-table [id=0000b2e0f05c4981a08ea717f386cb1b]) successfully deleted
I20260924 10:43:42.341923 20698 ts_tablet_manager.cc:1918] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:43:42.342149 20698 log.cc:1196] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/d218251c265848c7933942c5cc28bbf6
I20260924 10:43:42.342541 20698 ts_tablet_manager.cc:1939] T d218251c265848c7933942c5cc28bbf6 P 7e5dfe03173e4cada692b83e72f82202: Deleting consensus metadata
I20260924 10:43:42.342542 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 20394
I20260924 10:43:42.344240 20182 catalog_manager.cc:4841] TS 7e5dfe03173e4cada692b83e72f82202 (127.2.8.3:41701): tablet d218251c265848c7933942c5cc28bbf6 (table test-table [id=0000b2e0f05c4981a08ea717f386cb1b]) successfully deleted
I20260924 10:43:42.345244 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.344899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:44701 (local address 127.0.0.1:37920)
0924 10:43:42.345083 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:42.345157 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:44701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 10:43:42.345423 20185 catalog_manager.cc:4584] TS 92c12fa4af654548aa9ee7e5aa865570 (127.2.8.1:44701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d218251c265848c7933942c5cc28bbf6: Network error: Client connection negotiation failed: client connection to 127.2.8.1:44701: connect: Connection refused (error 111)
I20260924 10:43:42.348367 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 20528
I20260924 10:43:42.353914 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 20166
2026-09-24T10:43:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2422 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T10:43:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:43:42Z Disabled control of system clock
I20260924 10:43:42.384438 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:43773
--webserver_interface=127.2.8.62
--webserver_port=0
--builtin_ntp_servers=127.2.8.20:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:43773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:43:42.498836 20704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:42.499104 20704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:42.499137 20704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:42.503237 20704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:43:42.503298 20704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:42.503314 20704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:42.503338 20704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:43:42.503353 20704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:43:42.508059 20704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:43773
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:43773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:42.509112 20704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:42.510520 20704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:42.519484 20712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:42.519542 20709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:42.519616 20704 server_base.cc:1034] running on GCE node
W20260924 10:43:42.519542 20710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:42.520388 20704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:42.521652 20704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:42.522889 20704 hybrid_clock.cc:648] HybridClock initialized: now 1790246622522869 us; error 35 us; skew 500 ppm
I20260924 10:43:42.524946 20704 webserver.cc:460] Webserver started at http://127.2.8.62:44833/ using document root <none> and password file <none>
I20260924 10:43:42.525626 20704 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:42.525729 20704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:42.525969 20704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:42.528128 20704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "34ff19128e294fca8aa6548721fa7f2f"
format_stamp: "Formatted at 2026-09-24 10:43:42 on dist-test-slave-j38h"
server_key: "d9a1ca7101ef421fdb72ca21f736268f"
server_key_iv: "98a48562c75fa09d5027ac5f3bec0774"
server_key_version: "encryptionkey@0"
I20260924 10:43:42.528724 20704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "34ff19128e294fca8aa6548721fa7f2f"
format_stamp: "Formatted at 2026-09-24 10:43:42 on dist-test-slave-j38h"
server_key: "d9a1ca7101ef421fdb72ca21f736268f"
server_key_iv: "98a48562c75fa09d5027ac5f3bec0774"
server_key_version: "encryptionkey@0"
I20260924 10:43:42.532807 20704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:43:42.535349 20718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:42.536583 20704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:43:42.536825 20704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "34ff19128e294fca8aa6548721fa7f2f"
format_stamp: "Formatted at 2026-09-24 10:43:42 on dist-test-slave-j38h"
server_key: "d9a1ca7101ef421fdb72ca21f736268f"
server_key_iv: "98a48562c75fa09d5027ac5f3bec0774"
server_key_version: "encryptionkey@0"
I20260924 10:43:42.536952 20704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:42.558946 20704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:42.559715 20704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:42.559930 20704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:42.567360 20704 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:43773
I20260924 10:43:42.567399 20770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:43773 every 8 connection(s)
I20260924 10:43:42.568325 20704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:43:42.570825 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 20704
I20260924 10:43:42.570989 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:43:42.571295 2080 external_mini_cluster.cc:1428] Setting key f38be05b2bc56835f158e00bdd1c0ca5
I20260924 10:43:42.571525 20771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:42.577783 20771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f: Bootstrap starting.
I20260924 10:43:42.580129 20771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:42.580734 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.572709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.2.8.62:43773)
0924 10:43:42.573388 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:42.573403 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:42.573436 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:43:42.574241 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:42.574259 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:42.574671 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:42.575007 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:42.575473 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.576440 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.577391 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.577907 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.578083 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.578965 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:42.578970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:42.578983 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:42.579358 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:42.579410 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:42.579429 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:42.579550 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:42.579701 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 10:43:42.579954 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":167,"threads_started":1}
I20260924 10:43:42.580917 20771 log.cc:824] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:42.583146 20771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f: No bootstrap required, opened a new log
I20260924 10:43:42.586823 20771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ff19128e294fca8aa6548721fa7f2f" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43773 } }
I20260924 10:43:42.587118 20771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:42.587189 20771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ff19128e294fca8aa6548721fa7f2f, State: Initialized, Role: FOLLOWER
I20260924 10:43:42.587647 20771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ff19128e294fca8aa6548721fa7f2f" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43773 } }
I20260924 10:43:42.587812 20771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:43:42.587881 20771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:43:42.587982 20771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:42.588892 20771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ff19128e294fca8aa6548721fa7f2f" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43773 } }
I20260924 10:43:42.589253 20771 leader_election.cc:302] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34ff19128e294fca8aa6548721fa7f2f; no voters:
I20260924 10:43:42.589560 20771 leader_election.cc:288] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:43:42.589682 20776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:42.589932 20776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [term 1 LEADER]: Becoming Leader. State: Replica: 34ff19128e294fca8aa6548721fa7f2f, State: Running, Role: LEADER
I20260924 10:43:42.590240 20776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ff19128e294fca8aa6548721fa7f2f" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43773 } }
I20260924 10:43:42.590942 20771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:43:42.591991 20778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34ff19128e294fca8aa6548721fa7f2f. Latest consensus state: current_term: 1 leader_uuid: "34ff19128e294fca8aa6548721fa7f2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ff19128e294fca8aa6548721fa7f2f" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43773 } } }
I20260924 10:43:42.592118 20778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:42.592496 20785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:43:42.592469 20777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34ff19128e294fca8aa6548721fa7f2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ff19128e294fca8aa6548721fa7f2f" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43773 } } }
I20260924 10:43:42.592657 20777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:42.595674 20785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:43:42.601354 20785 catalog_manager.cc:1347] Generated new cluster ID: 1e64aca29c4540c8ab23959e7871356d
I20260924 10:43:42.601428 20785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:43:42.625290 20785 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:43:42.626191 20785 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:43:42.641939 20785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f: Generated new TSK 0
I20260924 10:43:42.642728 20785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:43:42.658505 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43773
--builtin_ntp_servers=127.2.8.20:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:43:42.770228 20795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:42.770499 20795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:42.770538 20795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:42.774523 20795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:42.774573 20795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:42.774645 20795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:43:42.779593 20795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:42.780707 20795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:42.782156 20795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:42.791898 20800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:42.791988 20801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:42.792069 20795 server_base.cc:1034] running on GCE node
W20260924 10:43:42.792197 20803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:42.792640 20795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:42.793342 20795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:42.794493 20795 hybrid_clock.cc:648] HybridClock initialized: now 1790246622794476 us; error 28 us; skew 500 ppm
I20260924 10:43:42.796316 20795 webserver.cc:460] Webserver started at http://127.2.8.1:45891/ using document root <none> and password file <none>
I20260924 10:43:42.796893 20795 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:42.796957 20795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:42.797240 20795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:42.799273 20795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "938b311056ae452aa561a01a8a66f605"
format_stamp: "Formatted at 2026-09-24 10:43:42 on dist-test-slave-j38h"
server_key: "78bbc0cb48e83e8d506828319c5b2410"
server_key_iv: "010b295c84e8c2b55163f9bb2e102349"
server_key_version: "encryptionkey@0"
I20260924 10:43:42.799860 20795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "938b311056ae452aa561a01a8a66f605"
format_stamp: "Formatted at 2026-09-24 10:43:42 on dist-test-slave-j38h"
server_key: "78bbc0cb48e83e8d506828319c5b2410"
server_key_iv: "010b295c84e8c2b55163f9bb2e102349"
server_key_version: "encryptionkey@0"
I20260924 10:43:42.803591 20795 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:43:42.805996 20809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:42.807106 20795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:43:42.807224 20795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "938b311056ae452aa561a01a8a66f605"
format_stamp: "Formatted at 2026-09-24 10:43:42 on dist-test-slave-j38h"
server_key: "78bbc0cb48e83e8d506828319c5b2410"
server_key_iv: "010b295c84e8c2b55163f9bb2e102349"
server_key_version: "encryptionkey@0"
I20260924 10:43:42.807327 20795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:42.837580 20795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:42.838471 20795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:42.838657 20795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:42.839288 20795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:42.840273 20795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:42.840322 20795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:42.840361 20795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:42.840376 20795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:42.849246 20795 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:40889
I20260924 10:43:42.849264 20922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:40889 every 8 connection(s)
I20260924 10:43:42.850153 20795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:43:42.855463 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 20795
I20260924 10:43:42.855602 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:43:42.855955 2080 external_mini_cluster.cc:1428] Setting key 5291eae162c214a77a42021bb6710e3a
I20260924 10:43:42.858348 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43773
--builtin_ntp_servers=127.2.8.20:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:43:42.862080 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.852187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:49255 (local address 127.2.8.62:43773)
0924 10:43:42.852363 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:42.852368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:42.853802 (+ 1434us) server_negotiation.cc:407] Connection header received
0924 10:43:42.855943 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:42.855948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:42.856050 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:42.856149 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:42.857390 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.858167 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.859996 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.860245 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.860360 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.861606 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:42.861609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:42.861611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:42.861624 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:42.861643 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:42.861650 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:42.861708 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:42.861904 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 10:43:42.861962 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 10:43:42.863359 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:42.852695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43773 (local address 127.2.8.1:49255)
0924 10:43:42.853603 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:42.853671 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:42.855625 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:42.856368 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:42.856394 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:42.856519 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:42.857238 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:42.857257 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.858339 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.858342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:42.859874 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:42.859888 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:42.860354 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:42.860412 (+ 58us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:42.860589 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:42.861280 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:43:42.861284 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:42.861343 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:43:42.861431 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:42.861734 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:42.861745 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:42.861754 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 10:43:42.861885 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 10:43:42.862207 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":168,"threads_started":1}
I20260924 10:43:42.864701 20923 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43773
I20260924 10:43:42.865010 20923 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:42.865687 20923 heartbeater.cc:511] Master 127.2.8.62:43773 requested a full tablet report, sending...
I20260924 10:43:42.867375 20735 ts_manager.cc:194] Registered new tserver with Master: 938b311056ae452aa561a01a8a66f605 (127.2.8.1:40889)
I20260924 10:43:42.868613 20735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.8.1:49255
W20260924 10:43:42.970927 20926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:42.971177 20926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:42.971215 20926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:42.975368 20926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:42.975423 20926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:42.975477 20926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:43:42.980155 20926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:42.981217 20926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:42.982570 20926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:42.992492 20932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:42.992522 20934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:42.992538 20931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:42.992877 20926 server_base.cc:1034] running on GCE node
I20260924 10:43:42.993312 20926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:42.993942 20926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:42.995126 20926 hybrid_clock.cc:648] HybridClock initialized: now 1790246622995077 us; error 69 us; skew 500 ppm
I20260924 10:43:42.997128 20926 webserver.cc:460] Webserver started at http://127.2.8.2:36459/ using document root <none> and password file <none>
I20260924 10:43:42.997695 20926 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:42.997794 20926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:42.998025 20926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:43.000052 20926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "e8ddb189b52f4ee2a172fec06fde921b"
format_stamp: "Formatted at 2026-09-24 10:43:42 on dist-test-slave-j38h"
server_key: "960b21754e493a041d690e8a28060168"
server_key_iv: "1aeac13e05a4d0b943262e370797f9a4"
server_key_version: "encryptionkey@0"
I20260924 10:43:43.000620 20926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "e8ddb189b52f4ee2a172fec06fde921b"
format_stamp: "Formatted at 2026-09-24 10:43:42 on dist-test-slave-j38h"
server_key: "960b21754e493a041d690e8a28060168"
server_key_iv: "1aeac13e05a4d0b943262e370797f9a4"
server_key_version: "encryptionkey@0"
I20260924 10:43:43.004326 20926 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.003s
I20260924 10:43:43.006880 20940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:43.008093 20926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:43:43.008230 20926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "e8ddb189b52f4ee2a172fec06fde921b"
format_stamp: "Formatted at 2026-09-24 10:43:42 on dist-test-slave-j38h"
server_key: "960b21754e493a041d690e8a28060168"
server_key_iv: "1aeac13e05a4d0b943262e370797f9a4"
server_key_version: "encryptionkey@0"
I20260924 10:43:43.008337 20926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:43.021209 20926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:43.021811 20926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:43.021996 20926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:43.022531 20926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:43.023561 20926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:43.023629 20926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:43.023690 20926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:43.023722 20926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:43.032902 20926 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:42493
I20260924 10:43:43.032933 21053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:42493 every 8 connection(s)
I20260924 10:43:43.033787 20926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:43:43.035384 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 20926
I20260924 10:43:43.035511 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:43:43.035853 2080 external_mini_cluster.cc:1428] Setting key bc210b5f6463102e374324a0022c2b42
I20260924 10:43:43.037945 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43773
--builtin_ntp_servers=127.2.8.20:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:43:43.044410 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.035877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:41619 (local address 127.2.8.62:43773)
0924 10:43:43.036043 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:43.036053 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:43.037367 (+ 1314us) server_negotiation.cc:407] Connection header received
0924 10:43:43.038898 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:43.038902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:43.038958 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:43.039038 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:43.039922 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.040524 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.042341 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.042565 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.042651 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:43.043954 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:43.043957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:43.043965 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:43.043977 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:43.043993 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:43.044001 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:43.044057 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:43.044254 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 10:43:43.044301 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 10:43:43.045694 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.036369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43773 (local address 127.2.8.2:41619)
0924 10:43:43.037219 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:43.037273 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:43.038694 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:43.039209 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:43.039223 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:43.039307 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:43.039778 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:43.039789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.040648 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.040655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:43.042219 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:43.042232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.042670 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.042734 (+ 64us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:43.042976 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:43.043644 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:43:43.043648 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:43.043708 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:43:43.043797 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:43.044088 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:43.044098 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:43.044106 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 10:43:43.044243 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 10:43:43.044567 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":138,"threads_started":1}
I20260924 10:43:43.046885 21054 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43773
I20260924 10:43:43.047123 21054 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:43.047627 21054 heartbeater.cc:511] Master 127.2.8.62:43773 requested a full tablet report, sending...
I20260924 10:43:43.048663 20735 ts_manager.cc:194] Registered new tserver with Master: e8ddb189b52f4ee2a172fec06fde921b (127.2.8.2:42493)
I20260924 10:43:43.049281 20735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.8.2:41619
W20260924 10:43:43.152446 21057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:43.152750 21057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:43.152823 21057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:43.156867 21057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:43.156961 21057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:43.157070 21057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:43:43.161887 21057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:43.162961 21057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:43.164366 21057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:43.174026 21065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:43.174021 21057 server_base.cc:1034] running on GCE node
W20260924 10:43:43.174006 21063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:43.173980 21062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:43.174613 21057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:43.175251 21057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:43.176497 21057 hybrid_clock.cc:648] HybridClock initialized: now 1790246623176477 us; error 34 us; skew 500 ppm
I20260924 10:43:43.178526 21057 webserver.cc:460] Webserver started at http://127.2.8.3:42163/ using document root <none> and password file <none>
I20260924 10:43:43.179128 21057 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:43.179224 21057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:43.179466 21057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:43.181696 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "8a50291e13d9423fa49bba6520e33831"
format_stamp: "Formatted at 2026-09-24 10:43:43 on dist-test-slave-j38h"
server_key: "82c272ec1fdce8904eb553e84854d30a"
server_key_iv: "57807c348f9403c5f509552bc24a1ab2"
server_key_version: "encryptionkey@0"
I20260924 10:43:43.182247 21057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "8a50291e13d9423fa49bba6520e33831"
format_stamp: "Formatted at 2026-09-24 10:43:43 on dist-test-slave-j38h"
server_key: "82c272ec1fdce8904eb553e84854d30a"
server_key_iv: "57807c348f9403c5f509552bc24a1ab2"
server_key_version: "encryptionkey@0"
I20260924 10:43:43.185959 21057 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:43:43.188346 21071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:43.189661 21057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:43:43.189764 21057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "8a50291e13d9423fa49bba6520e33831"
format_stamp: "Formatted at 2026-09-24 10:43:43 on dist-test-slave-j38h"
server_key: "82c272ec1fdce8904eb553e84854d30a"
server_key_iv: "57807c348f9403c5f509552bc24a1ab2"
server_key_version: "encryptionkey@0"
I20260924 10:43:43.189847 21057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:43.204047 21057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:43.204630 21057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:43.204793 21057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:43.205363 21057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:43.206254 21057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:43.206326 21057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:43.206386 21057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:43.206439 21057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:43.215022 21057 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:39445
I20260924 10:43:43.215062 21184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:39445 every 8 connection(s)
I20260924 10:43:43.215860 21057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:43:43.224829 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.217659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:52929 (local address 127.2.8.62:43773)
0924 10:43:43.217786 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:43.217790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:43.218857 (+ 1067us) server_negotiation.cc:407] Connection header received
0924 10:43:43.220240 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:43.220244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:43.220303 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:43.220407 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:43.221308 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.221838 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.222953 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.223184 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.223295 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:43.224330 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:43.224333 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:43.224338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:43.224355 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:43.224373 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:43.224381 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:43.224471 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:43.224627 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 10:43:43.224688 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:43:43.225227 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 21057
I20260924 10:43:43.225353 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:43:43.225625 2080 external_mini_cluster.cc:1428] Setting key a8e858c635f6c2ba649f79c2627ef920
I20260924 10:43:43.225782 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.218051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43773 (local address 127.2.8.3:52929)
0924 10:43:43.218712 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:43.218751 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:43.220027 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:43.220583 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:43.220589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:43.220671 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:43.221168 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:43.221182 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.221965 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.221975 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:43.222829 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:43.222838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.223282 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.223330 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:43.223494 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:43.224080 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:43:43.224082 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:43.224121 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:43:43.224179 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:43.224473 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:43.224485 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:43.224492 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 10:43:43.224616 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 10:43:43.224872 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260924 10:43:43.226919 21185 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43773
I20260924 10:43:43.227159 21185 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:43.227728 21185 heartbeater.cc:511] Master 127.2.8.62:43773 requested a full tablet report, sending...
I20260924 10:43:43.228660 20734 ts_manager.cc:194] Registered new tserver with Master: 8a50291e13d9423fa49bba6520e33831 (127.2.8.3:39445)
I20260924 10:43:43.229264 20734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.8.3:52929
I20260924 10:43:43.239358 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:43:43.245267 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.241258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.2.8.62:43773)
0924 10:43:43.241390 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:43.241394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:43.241648 (+ 254us) server_negotiation.cc:407] Connection header received
0924 10:43:43.241770 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:43.241774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:43.241823 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:43.241914 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:43.242202 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.242711 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.243561 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.243782 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.243899 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:43.244754 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:43.244757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:43.244759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:43.244770 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:43.244786 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:43.244793 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:43.244929 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:43.245099 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 10:43:43.245137 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 10:43:43.247823 20734 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:43:43.250851 20734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:43:43.265021 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.258970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:40889 (local address 127.0.0.1:54266)
0924 10:43:43.259566 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:43.259614 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:43.259731 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:43.260623 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:43.260629 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:43.260671 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:43.260832 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:43.260839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.262415 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.262418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:43.263537 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:43.263545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.264468 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.264471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:43.264569 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:43.264618 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 10:43:43.264711 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 10:43:43.264830 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":60,"spinlock_wait_cycles":7680,"thread_start_us":66,"threads_started":1}
I20260924 10:43:43.265057 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.259083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.2.8.1:40889)
0924 10:43:43.260084 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:43.260099 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:43.260125 (+ 26us) server_negotiation.cc:407] Connection header received
0924 10:43:43.260207 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:43.260212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:43.260461 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:43.260978 (+ 517us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:43.261137 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.262290 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.263648 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.264331 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.264517 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:43.264697 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:43.264752 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 10:43:43.264861 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":122,"threads_started":1}
I20260924 10:43:43.265362 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.259984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34160 (local address 127.2.8.2:42493)
0924 10:43:43.260405 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:43.260418 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:43.260776 (+ 358us) server_negotiation.cc:407] Connection header received
0924 10:43:43.260863 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:43.260880 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:43.261085 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:43.261234 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:43.261458 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.262523 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.263951 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.264722 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.264828 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:43.265006 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:43.265094 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 10:43:43.265190 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":115,"threads_started":1}
I20260924 10:43:43.265667 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.259043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:39445 (local address 127.0.0.1:34162)
0924 10:43:43.259949 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:43.259963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:43.260064 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:43.260625 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:43.260631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:43.260667 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:43.260840 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:43.260848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.261988 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.261994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:43.263436 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:43.263447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.264389 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.264393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:43.265359 (+ 966us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:43.265393 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 10:43:43.265467 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 10:43:43.265537 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":4,"spinlock_wait_cycles":8192,"thread_start_us":104,"threads_started":1}
I20260924 10:43:43.265672 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.259658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:42493 (local address 127.0.0.1:34160)
0924 10:43:43.260659 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:43.260673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:43.260763 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:43.261226 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:43.261229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:43.261248 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:43.261340 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:43.261346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.262631 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.262638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:43.263821 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:43.263832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.264840 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.264843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:43.264943 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:43.264964 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:43:43.265071 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 10:43:43.265579 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":21,"thread_start_us":82,"threads_started":1}
I20260924 10:43:43.266070 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.259503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.2.8.3:39445)
0924 10:43:43.259936 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:43.259950 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:43.260160 (+ 210us) server_negotiation.cc:407] Connection header received
0924 10:43:43.260228 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:43.260232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:43.260410 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:43.260574 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:43.260965 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.261884 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.263567 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.264256 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.264380 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:43.264567 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:43.265842 (+ 1275us) server_negotiation.cc:299] Negotiation successful
0924 10:43:43.265941 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":142,"threads_started":1}
I20260924 10:43:43.268397 20988 tablet_service.cc:1467] Processing CreateTablet for tablet 57a144b180e347cba401e6ffe0842f1b (DEFAULT_TABLE table=test-table [id=6dccd9e869934ea28e92425cb3bf69b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:43.268517 20857 tablet_service.cc:1467] Processing CreateTablet for tablet 57a144b180e347cba401e6ffe0842f1b (DEFAULT_TABLE table=test-table [id=6dccd9e869934ea28e92425cb3bf69b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:43.268860 21119 tablet_service.cc:1467] Processing CreateTablet for tablet 57a144b180e347cba401e6ffe0842f1b (DEFAULT_TABLE table=test-table [id=6dccd9e869934ea28e92425cb3bf69b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:43.269490 20988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a144b180e347cba401e6ffe0842f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:43.269584 20857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a144b180e347cba401e6ffe0842f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:43.269724 21119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a144b180e347cba401e6ffe0842f1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:43.275420 21202 tablet_bootstrap.cc:492] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b: Bootstrap starting.
I20260924 10:43:43.277282 21203 tablet_bootstrap.cc:492] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831: Bootstrap starting.
I20260924 10:43:43.277842 21202 tablet_bootstrap.cc:654] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:43.278720 21202 log.cc:824] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:43.279242 21203 tablet_bootstrap.cc:654] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:43.279345 21204 tablet_bootstrap.cc:492] T 57a144b180e347cba401e6ffe0842f1b P 938b311056ae452aa561a01a8a66f605: Bootstrap starting.
I20260924 10:43:43.280054 21203 log.cc:824] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:43.280874 21202 tablet_bootstrap.cc:492] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b: No bootstrap required, opened a new log
I20260924 10:43:43.281090 21202 ts_tablet_manager.cc:1397] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:43:43.281201 21204 tablet_bootstrap.cc:654] T 57a144b180e347cba401e6ffe0842f1b P 938b311056ae452aa561a01a8a66f605: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:43.281805 21204 log.cc:824] T 57a144b180e347cba401e6ffe0842f1b P 938b311056ae452aa561a01a8a66f605: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:43.282064 21203 tablet_bootstrap.cc:492] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831: No bootstrap required, opened a new log
I20260924 10:43:43.282248 21203 ts_tablet_manager.cc:1397] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:43:43.283383 21204 tablet_bootstrap.cc:492] T 57a144b180e347cba401e6ffe0842f1b P 938b311056ae452aa561a01a8a66f605: No bootstrap required, opened a new log
I20260924 10:43:43.283604 21204 ts_tablet_manager.cc:1397] T 57a144b180e347cba401e6ffe0842f1b P 938b311056ae452aa561a01a8a66f605: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260924 10:43:43.285442 21202 raft_consensus.cc:348] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938b311056ae452aa561a01a8a66f605" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 40889 } } peers { permanent_uuid: "8a50291e13d9423fa49bba6520e33831" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39445 } } peers { permanent_uuid: "e8ddb189b52f4ee2a172fec06fde921b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42493 } }
I20260924 10:43:43.285708 21202 raft_consensus.cc:374] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:43.285771 21202 raft_consensus.cc:729] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ddb189b52f4ee2a172fec06fde921b, State: Initialized, Role: FOLLOWER
I20260924 10:43:43.286229 21202 consensus_queue.cc:260] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938b311056ae452aa561a01a8a66f605" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 40889 } } peers { permanent_uuid: "8a50291e13d9423fa49bba6520e33831" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39445 } } peers { permanent_uuid: "e8ddb189b52f4ee2a172fec06fde921b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42493 } }
I20260924 10:43:43.286314 21203 raft_consensus.cc:348] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938b311056ae452aa561a01a8a66f605" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 40889 } } peers { permanent_uuid: "8a50291e13d9423fa49bba6520e33831" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39445 } } peers { permanent_uuid: "e8ddb189b52f4ee2a172fec06fde921b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42493 } }
I20260924 10:43:43.286551 21203 raft_consensus.cc:374] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:43.286612 21203 raft_consensus.cc:729] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a50291e13d9423fa49bba6520e33831, State: Initialized, Role: FOLLOWER
I20260924 10:43:43.286993 21054 heartbeater.cc:503] Master 127.2.8.62:43773 was elected leader, sending a full tablet report...
I20260924 10:43:43.287222 21202 ts_tablet_manager.cc:1428] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:43:43.287365 21204 raft_consensus.cc:348] T 57a144b180e347cba401e6ffe0842f1b P 938b311056ae452aa561a01a8a66f605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938b311056ae452aa561a01a8a66f605" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 40889 } } peers { permanent_uuid: "8a50291e13d9423fa49bba6520e33831" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39445 } } peers { permanent_uuid: "e8ddb189b52f4ee2a172fec06fde921b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42493 } }
I20260924 10:43:43.287572 21204 raft_consensus.cc:374] T 57a144b180e347cba401e6ffe0842f1b P 938b311056ae452aa561a01a8a66f605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:43.287631 21204 raft_consensus.cc:729] T 57a144b180e347cba401e6ffe0842f1b P 938b311056ae452aa561a01a8a66f605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938b311056ae452aa561a01a8a66f605, State: Initialized, Role: FOLLOWER
I20260924 10:43:43.288074 21203 consensus_queue.cc:260] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938b311056ae452aa561a01a8a66f605" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 40889 } } peers { permanent_uuid: "8a50291e13d9423fa49bba6520e33831" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39445 } } peers { permanent_uuid: "e8ddb189b52f4ee2a172fec06fde921b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42493 } }
I20260924 10:43:43.288069 21204 consensus_queue.cc:260] T 57a144b180e347cba401e6ffe0842f1b P 938b311056ae452aa561a01a8a66f605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938b311056ae452aa561a01a8a66f605" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 40889 } } peers { permanent_uuid: "8a50291e13d9423fa49bba6520e33831" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39445 } } peers { permanent_uuid: "e8ddb189b52f4ee2a172fec06fde921b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42493 } }
I20260924 10:43:43.288733 21185 heartbeater.cc:503] Master 127.2.8.62:43773 was elected leader, sending a full tablet report...
I20260924 10:43:43.289000 21203 ts_tablet_manager.cc:1428] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 10:43:43.289003 21204 ts_tablet_manager.cc:1428] T 57a144b180e347cba401e6ffe0842f1b P 938b311056ae452aa561a01a8a66f605: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 10:43:43.289330 20923 heartbeater.cc:503] Master 127.2.8.62:43773 was elected leader, sending a full tablet report...
W20260924 10:43:43.351354 20924 tablet.cc:2369] T 57a144b180e347cba401e6ffe0842f1b P 938b311056ae452aa561a01a8a66f605: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:43:43.411288 21208 raft_consensus.cc:482] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:43:43.411620 21208 raft_consensus.cc:504] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938b311056ae452aa561a01a8a66f605" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 40889 } } peers { permanent_uuid: "8a50291e13d9423fa49bba6520e33831" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39445 } } peers { permanent_uuid: "e8ddb189b52f4ee2a172fec06fde921b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42493 } }
I20260924 10:43:43.413121 21208 leader_election.cc:288] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 938b311056ae452aa561a01a8a66f605 (127.2.8.1:40889), 8a50291e13d9423fa49bba6520e33831 (127.2.8.3:39445)
I20260924 10:43:43.416952 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.413276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:40889 (local address 127.2.8.2:49617)
0924 10:43:43.413408 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:43.413423 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:43.413509 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:43.414093 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:43.414097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:43.414118 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:43.414269 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:43.414281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.415039 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.415046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:43.415859 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:43.415868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.416506 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.416512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:43.416608 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:43.416678 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 10:43:43.416737 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 10:43:43.416793 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":22}
I20260924 10:43:43.417011 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.413768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:49617 (local address 127.2.8.1:40889)
0924 10:43:43.413894 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:43.413897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:43.413909 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:43:43.413953 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:43.413955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:43.413992 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:43.414054 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:43.414404 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.414928 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.415985 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.416385 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.416466 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:43.416547 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:43.416783 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 10:43:43.416861 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:43:43.417073 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.413400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:39445 (local address 127.2.8.2:47545)
0924 10:43:43.413711 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:43.413722 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:43.413830 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:43.414163 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:43.414166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:43.414186 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:43.414355 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:43.414362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.415102 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.415104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:43.415868 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:43.415884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.416605 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.416608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:43.416706 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:43.416750 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 10:43:43.416830 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 10:43:43.416896 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":66,"thread_start_us":114,"threads_started":1}
I20260924 10:43:43.417091 21199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.413746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:47545 (local address 127.2.8.3:39445)
0924 10:43:43.413914 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:43.413919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:43.413932 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:43:43.413988 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:43.413990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:43.414025 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:43.414119 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:43.414478 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.414968 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.416027 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.416449 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.416555 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:43.416616 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:43.416869 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 10:43:43.416939 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 10:43:43.417649 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57a144b180e347cba401e6ffe0842f1b" candidate_uuid: "e8ddb189b52f4ee2a172fec06fde921b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "938b311056ae452aa561a01a8a66f605" is_pre_election: true
I20260924 10:43:43.417783 21139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57a144b180e347cba401e6ffe0842f1b" candidate_uuid: "e8ddb189b52f4ee2a172fec06fde921b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a50291e13d9423fa49bba6520e33831" is_pre_election: true
I20260924 10:43:43.417981 20877 raft_consensus.cc:2454] T 57a144b180e347cba401e6ffe0842f1b P 938b311056ae452aa561a01a8a66f605 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8ddb189b52f4ee2a172fec06fde921b in term 0.
I20260924 10:43:43.418121 21139 raft_consensus.cc:2454] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8ddb189b52f4ee2a172fec06fde921b in term 0.
I20260924 10:43:43.418526 20943 leader_election.cc:302] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8a50291e13d9423fa49bba6520e33831, 938b311056ae452aa561a01a8a66f605, e8ddb189b52f4ee2a172fec06fde921b; no voters:
I20260924 10:43:43.418795 21208 raft_consensus.cc:2781] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:43:43.418871 21208 raft_consensus.cc:482] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:43:43.418905 21208 raft_consensus.cc:3037] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:43.419932 21208 raft_consensus.cc:504] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938b311056ae452aa561a01a8a66f605" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 40889 } } peers { permanent_uuid: "8a50291e13d9423fa49bba6520e33831" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39445 } } peers { permanent_uuid: "e8ddb189b52f4ee2a172fec06fde921b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42493 } }
I20260924 10:43:43.420329 21208 leader_election.cc:288] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [CANDIDATE]: Term 1 election: Requested vote from peers 938b311056ae452aa561a01a8a66f605 (127.2.8.1:40889), 8a50291e13d9423fa49bba6520e33831 (127.2.8.3:39445)
I20260924 10:43:43.420701 21139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57a144b180e347cba401e6ffe0842f1b" candidate_uuid: "e8ddb189b52f4ee2a172fec06fde921b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a50291e13d9423fa49bba6520e33831"
I20260924 10:43:43.420706 20877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57a144b180e347cba401e6ffe0842f1b" candidate_uuid: "e8ddb189b52f4ee2a172fec06fde921b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "938b311056ae452aa561a01a8a66f605"
I20260924 10:43:43.420809 21139 raft_consensus.cc:3037] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:43.420862 20877 raft_consensus.cc:3037] T 57a144b180e347cba401e6ffe0842f1b P 938b311056ae452aa561a01a8a66f605 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:43.421878 21139 raft_consensus.cc:2454] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8ddb189b52f4ee2a172fec06fde921b in term 1.
I20260924 10:43:43.421970 20877 raft_consensus.cc:2454] T 57a144b180e347cba401e6ffe0842f1b P 938b311056ae452aa561a01a8a66f605 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8ddb189b52f4ee2a172fec06fde921b in term 1.
I20260924 10:43:43.422206 20944 leader_election.cc:302] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a50291e13d9423fa49bba6520e33831, e8ddb189b52f4ee2a172fec06fde921b; no voters:
I20260924 10:43:43.422376 21208 raft_consensus.cc:2781] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:43.422627 21208 raft_consensus.cc:686] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [term 1 LEADER]: Becoming Leader. State: Replica: e8ddb189b52f4ee2a172fec06fde921b, State: Running, Role: LEADER
I20260924 10:43:43.423004 21208 consensus_queue.cc:237] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938b311056ae452aa561a01a8a66f605" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 40889 } } peers { permanent_uuid: "8a50291e13d9423fa49bba6520e33831" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39445 } } peers { permanent_uuid: "e8ddb189b52f4ee2a172fec06fde921b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42493 } }
I20260924 10:43:43.426218 20735 catalog_manager.cc:5495] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b reported cstate change: term changed from 0 to 1, leader changed from <none> to e8ddb189b52f4ee2a172fec06fde921b (127.2.8.2). New cstate: current_term: 1 leader_uuid: "e8ddb189b52f4ee2a172fec06fde921b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "938b311056ae452aa561a01a8a66f605" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 40889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a50291e13d9423fa49bba6520e33831" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8ddb189b52f4ee2a172fec06fde921b" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42493 } health_report { overall_health: HEALTHY } } }
I20260924 10:43:43.458881 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.454705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34174 (local address 127.2.8.2:42493)
0924 10:43:43.454898 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:43.454902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:43.454916 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:43:43.454988 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:43.454991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:43.455044 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:43.455188 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:43:43.455514 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.456408 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.457317 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:43.457549 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:43.457659 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:43.457994 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:43:43.458398 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:43:43.458559 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:43.458668 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 10:43:43.458707 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260924 10:43:43.465220 21139 raft_consensus.cc:1261] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831 [term 1 FOLLOWER]: Refusing update from remote peer e8ddb189b52f4ee2a172fec06fde921b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:43:43.465220 20877 raft_consensus.cc:1261] T 57a144b180e347cba401e6ffe0842f1b P 938b311056ae452aa561a01a8a66f605 [term 1 FOLLOWER]: Refusing update from remote peer e8ddb189b52f4ee2a172fec06fde921b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:43:43.466069 21208 consensus_queue.cc:1035] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a50291e13d9423fa49bba6520e33831" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 39445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:43:43.466382 21212 consensus_queue.cc:1035] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [LEADER]: Connected to new peer: Peer: permanent_uuid: "938b311056ae452aa561a01a8a66f605" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 40889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 10:43:43.466747 21186 tablet.cc:2369] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:43:43.474982 21215 mvcc.cc:205] Tried to move back new op lower bound from 7332850169704488960 to 7332850169543184384. Current Snapshot: MvccSnapshot[applied={T|T < 7332850169704488960}]
I20260924 10:43:43.484179 20735 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:43:43.484392 20735 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:43:43.486821 20735 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34ff19128e294fca8aa6548721fa7f2f: Sending DeleteTablet for 3 replicas of tablet 57a144b180e347cba401e6ffe0842f1b
I20260924 10:43:43.487730 20857 tablet_service.cc:1514] Processing DeleteTablet for tablet 57a144b180e347cba401e6ffe0842f1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:43 UTC) from {username='slave'} at 127.0.0.1:54266
I20260924 10:43:43.487944 20988 tablet_service.cc:1514] Processing DeleteTablet for tablet 57a144b180e347cba401e6ffe0842f1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:43 UTC) from {username='slave'} at 127.0.0.1:34160
I20260924 10:43:43.488251 21119 tablet_service.cc:1514] Processing DeleteTablet for tablet 57a144b180e347cba401e6ffe0842f1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:43 UTC) from {username='slave'} at 127.0.0.1:34162
I20260924 10:43:43.488389 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 20795
I20260924 10:43:43.488531 21224 tablet_replica.cc:330] T 57a144b180e347cba401e6ffe0842f1b P 938b311056ae452aa561a01a8a66f605: stopping tablet replica
I20260924 10:43:43.489064 21225 tablet_replica.cc:330] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b: stopping tablet replica
I20260924 10:43:43.489352 21225 raft_consensus.cc:2229] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:43:43.489368 21226 tablet_replica.cc:330] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831: stopping tablet replica
I20260924 10:43:43.489647 21226 raft_consensus.cc:2229] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:43:43.489713 21225 raft_consensus.cc:2258] T 57a144b180e347cba401e6ffe0842f1b P e8ddb189b52f4ee2a172fec06fde921b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:43:43.489957 21226 raft_consensus.cc:2258] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:43:43.491822 21226 ts_tablet_manager.cc:1905] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:43:43.495131 21226 ts_tablet_manager.cc:1918] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:43:43.495234 21226 log.cc:1196] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/57a144b180e347cba401e6ffe0842f1b
I20260924 10:43:43.495554 21226 ts_tablet_manager.cc:1939] T 57a144b180e347cba401e6ffe0842f1b P 8a50291e13d9423fa49bba6520e33831: Deleting consensus metadata
W20260924 10:43:43.497020 20721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.8.1:40889 (error 108)
I20260924 10:43:43.497051 20722 catalog_manager.cc:4841] TS 8a50291e13d9423fa49bba6520e33831 (127.2.8.3:39445): tablet 57a144b180e347cba401e6ffe0842f1b (table test-table [id=6dccd9e869934ea28e92425cb3bf69b1]) successfully deleted
I20260924 10:43:43.498009 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 20926
I20260924 10:43:43.498231 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.497902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:40889 (local address 127.0.0.1:54272)
0924 10:43:43.498089 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:43.498151 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:40889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260924 10:43:43.499367 20721 catalog_manager.cc:4584] TS 938b311056ae452aa561a01a8a66f605 (127.2.8.1:40889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57a144b180e347cba401e6ffe0842f1b: Network error: Client connection negotiation failed: client connection to 127.2.8.1:40889: connect: Connection refused (error 111)
I20260924 10:43:43.503875 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 21057
I20260924 10:43:43.505944 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:43.505701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:42493 (local address 127.0.0.1:34178)
0924 10:43:43.505827 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:43.505876 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.2:42493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 10:43:43.509654 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 20704
2026-09-24T10:43:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1154 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:43:43 dist-test-slave-j38h krb5kdc[21230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:43:43 dist-test-slave-j38h krb5kdc[21230](info): set up 2 sockets
Sep 24 10:43:43 dist-test-slave-j38h krb5kdc[21230](info): commencing operation
krb5kdc: starting...
W20260924 10:43:44.620954 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:43:44 dist-test-slave-j38h krb5kdc[21230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:43:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:43:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:43:44.745242 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:43219
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:43219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:44.856119 21243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:44.856540 21243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:44.856664 21243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:44.862221 21243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:43:44.862291 21243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:44.862310 21243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:44.862349 21243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:43:44.862380 21243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:43:44.867909 21243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:43219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:43219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:44.869055 21243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:44.870102 21243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:44.878700 21249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:44.878790 21243 server_base.cc:1034] running on GCE node
W20260924 10:43:44.878665 21248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:44.878950 21251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:44.879559 21243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:44.880725 21243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:44.881922 21243 hybrid_clock.cc:648] HybridClock initialized: now 1790246624881898 us; error 47 us; skew 500 ppm
Sep 24 10:43:44 dist-test-slave-j38h krb5kdc[21230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246624, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:44.884660 21243 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:43:44.885550 21243 webserver.cc:460] Webserver started at http://127.2.8.62:35195/ using document root <none> and password file <none>
I20260924 10:43:44.886056 21243 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:44.886114 21243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:44.886281 21243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:44.888240 21243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "bead6555fa7348e1b2fd5a9f8ce8f008"
format_stamp: "Formatted at 2026-09-24 10:43:44 on dist-test-slave-j38h"
server_key: "fd4591dade9a424cda9e374c14140fc7"
server_key_iv: "5c540e6c644e7a75766b8a178cf2a78f"
server_key_version: "encryptionkey@0"
I20260924 10:43:44.888756 21243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "bead6555fa7348e1b2fd5a9f8ce8f008"
format_stamp: "Formatted at 2026-09-24 10:43:44 on dist-test-slave-j38h"
server_key: "fd4591dade9a424cda9e374c14140fc7"
server_key_iv: "5c540e6c644e7a75766b8a178cf2a78f"
server_key_version: "encryptionkey@0"
I20260924 10:43:44.892393 21243 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:43:44.894824 21258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:44.896164 21243 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:43:44.896371 21243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "bead6555fa7348e1b2fd5a9f8ce8f008"
format_stamp: "Formatted at 2026-09-24 10:43:44 on dist-test-slave-j38h"
server_key: "fd4591dade9a424cda9e374c14140fc7"
server_key_iv: "5c540e6c644e7a75766b8a178cf2a78f"
server_key_version: "encryptionkey@0"
I20260924 10:43:44.896478 21243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:44.917009 21243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:44.920085 21243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:44.920316 21243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:44.927917 21243 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:43219
I20260924 10:43:44.927942 21310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:43219 every 8 connection(s)
I20260924 10:43:44.928936 21243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:43:44.931860 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 21243
I20260924 10:43:44.931898 21311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:44.932016 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:43:44.932327 2080 external_mini_cluster.cc:1428] Setting key d76fbbf0f4b06866f0b41d663e3e25ed
I20260924 10:43:44.938493 21311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008: Bootstrap starting.
I20260924 10:43:44.941109 21311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008: Neither blocks nor log segments found. Creating new log.
Sep 24 10:43:44 dist-test-slave-j38h krb5kdc[21230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:43:44.941891 21311 log.cc:824] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:44.943748 21311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008: No bootstrap required, opened a new log
I20260924 10:43:44.946697 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:44.933762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.2.8.62:43219)
0924 10:43:44.934212 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:44.934226 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:44.934261 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:43:44.935159 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:44.935179 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:44.935527 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:44.935909 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:44.936688 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:44.937751 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:44.938658 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:44.939319 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:44.939481 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:44.942034 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:44.942040 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:44.942055 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:44.942085 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:44.943877 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:44.944361 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:44.944365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:44.944369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:44.944490 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:44.944777 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:44.944779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:44.944780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:44.945233 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:44.945376 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:44.945663 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 10:43:44.945971 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":147,"threads_started":1}
I20260924 10:43:44.947888 21311 raft_consensus.cc:348] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead6555fa7348e1b2fd5a9f8ce8f008" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43219 } }
I20260924 10:43:44.948132 21311 raft_consensus.cc:374] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:44.948211 21311 raft_consensus.cc:729] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bead6555fa7348e1b2fd5a9f8ce8f008, State: Initialized, Role: FOLLOWER
I20260924 10:43:44.948638 21311 consensus_queue.cc:260] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead6555fa7348e1b2fd5a9f8ce8f008" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43219 } }
I20260924 10:43:44.948788 21311 raft_consensus.cc:388] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:43:44.948889 21311 raft_consensus.cc:482] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:43:44.949034 21311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:44.949977 21311 raft_consensus.cc:504] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead6555fa7348e1b2fd5a9f8ce8f008" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43219 } }
I20260924 10:43:44.950316 21311 leader_election.cc:302] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bead6555fa7348e1b2fd5a9f8ce8f008; no voters:
I20260924 10:43:44.950629 21311 leader_election.cc:288] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:43:44.950780 21316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:44.951011 21316 raft_consensus.cc:686] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [term 1 LEADER]: Becoming Leader. State: Replica: bead6555fa7348e1b2fd5a9f8ce8f008, State: Running, Role: LEADER
I20260924 10:43:44.951620 21316 consensus_queue.cc:237] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead6555fa7348e1b2fd5a9f8ce8f008" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43219 } }
I20260924 10:43:44.951907 21311 sys_catalog.cc:564] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:43:44.953461 21318 sys_catalog.cc:455] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bead6555fa7348e1b2fd5a9f8ce8f008. Latest consensus state: current_term: 1 leader_uuid: "bead6555fa7348e1b2fd5a9f8ce8f008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead6555fa7348e1b2fd5a9f8ce8f008" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43219 } } }
I20260924 10:43:44.953467 21317 sys_catalog.cc:455] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bead6555fa7348e1b2fd5a9f8ce8f008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bead6555fa7348e1b2fd5a9f8ce8f008" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43219 } } }
I20260924 10:43:44.953604 21318 sys_catalog.cc:458] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:44.953605 21317 sys_catalog.cc:458] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008 [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:44.953989 21330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:43:44.956859 21330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:43:44.962661 21330 catalog_manager.cc:1347] Generated new cluster ID: 73c2e7f8a7f341c3b14b73465e2542be
I20260924 10:43:44.962755 21330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:43:44.970237 21330 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:43:44.971218 21330 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:43:44.982082 21330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008: Generated new TSK 0
I20260924 10:43:44.982846 21330 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.8.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
Entry for principal kudu/127.2.8.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab.
I20260924 10:43:45.018203 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43219
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:45.142870 21337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:45.143126 21337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:45.143168 21337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:45.147200 21337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:45.147266 21337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:45.147383 21337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:43:45.153206 21337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.1.keytab
--principal=kudu/127.2.8.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:45.154328 21337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:45.155301 21337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:45.164892 21343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:45.164911 21345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:45.165021 21342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:45.165158 21337 server_base.cc:1034] running on GCE node
I20260924 10:43:45.165633 21337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:45.166280 21337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:45.167443 21337 hybrid_clock.cc:648] HybridClock initialized: now 1790246625167436 us; error 36 us; skew 500 ppm
Sep 24 10:43:45 dist-test-slave-j38h krb5kdc[21230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246625, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:45.170256 21337 init.cc:377] Logged in from keytab as kudu/127.2.8.1@KRBTEST.COM (short username kudu)
I20260924 10:43:45.171172 21337 webserver.cc:460] Webserver started at http://127.2.8.1:34621/ using document root <none> and password file <none>
I20260924 10:43:45.171741 21337 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:45.171831 21337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:45.172083 21337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:45.174187 21337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "7e7dcfa6d3b0484cb6e92f1c4c94110f"
format_stamp: "Formatted at 2026-09-24 10:43:45 on dist-test-slave-j38h"
server_key: "686b59b3ed9ccdf8f8d5939a1f09dac2"
server_key_iv: "74663a1273dfaf50592d004f2c7768a7"
server_key_version: "encryptionkey@0"
I20260924 10:43:45.174815 21337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "7e7dcfa6d3b0484cb6e92f1c4c94110f"
format_stamp: "Formatted at 2026-09-24 10:43:45 on dist-test-slave-j38h"
server_key: "686b59b3ed9ccdf8f8d5939a1f09dac2"
server_key_iv: "74663a1273dfaf50592d004f2c7768a7"
server_key_version: "encryptionkey@0"
I20260924 10:43:45.178454 21337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:43:45.180851 21352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:45.182058 21337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:43:45.182193 21337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "7e7dcfa6d3b0484cb6e92f1c4c94110f"
format_stamp: "Formatted at 2026-09-24 10:43:45 on dist-test-slave-j38h"
server_key: "686b59b3ed9ccdf8f8d5939a1f09dac2"
server_key_iv: "74663a1273dfaf50592d004f2c7768a7"
server_key_version: "encryptionkey@0"
I20260924 10:43:45.182301 21337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:45.206681 21337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:45.209738 21337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:45.209952 21337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:45.210548 21337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:45.211352 21337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:45.211405 21337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:45.211473 21337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:45.211505 21337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:45.220104 21337 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:41431
I20260924 10:43:45.220124 21465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:41431 every 8 connection(s)
I20260924 10:43:45.221012 21337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:43:45.225271 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 21337
I20260924 10:43:45.225454 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:43:45.225745 2080 external_mini_cluster.cc:1428] Setting key 42417399c7b6e7d2d2ffb9b03523f0e8
Sep 24 10:43:45 dist-test-slave-j38h krb5kdc[21230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246625, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.1@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.2@KRBTEST.COM; defaulting to no policy
I20260924 10:43:45.235139 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.222744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:55061 (local address 127.2.8.62:43219)
0924 10:43:45.222891 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:45.222894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:45.223711 (+ 817us) server_negotiation.cc:407] Connection header received
0924 10:43:45.225065 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:45.225069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:45.225126 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:45.225234 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:45.226644 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.227279 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.228419 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.228653 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.228763 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.231510 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:45.231514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:45.231516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:45.231544 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:45.233104 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:45.233610 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:45.233616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:45.233618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:45.233699 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:45.234046 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:45.234051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:45.234052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:45.234236 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:45.234346 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:45.234841 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 10:43:45.234970 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":58}
I20260924 10:43:45.235946 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.223113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43219 (local address 127.2.8.1:55061)
0924 10:43:45.223552 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:45.223596 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:45.224812 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:45.225439 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:45.225449 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:45.225911 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:45.226500 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:45.226519 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.227402 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.227405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:45.228290 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:45.228299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.228750 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.228804 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:45.228997 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.229657 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:43:45.229681 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:45.231315 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:45.233222 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:45.233228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:45.233244 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:45.233504 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:45.233794 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:45.233798 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:45.233800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:45.233949 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:45.234344 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:45.234351 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:45.234610 (+ 259us) client_negotiation.cc:769] Sending connection context
0924 10:43:45.234804 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 10:43:45.235133 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":117,"threads_started":1}
I20260924 10:43:45.237201 21466 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43219
I20260924 10:43:45.237459 21466 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.2@KRBTEST.COM" created.
I20260924 10:43:45.238031 21466 heartbeater.cc:511] Master 127.2.8.62:43219 requested a full tablet report, sending...
I20260924 10:43:45.239765 21275 ts_manager.cc:194] Registered new tserver with Master: 7e7dcfa6d3b0484cb6e92f1c4c94110f (127.2.8.1:41431)
I20260924 10:43:45.241516 21275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.1@KRBTEST.COM'} at 127.2.8.1:55061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
Entry for principal kudu/127.2.8.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab.
I20260924 10:43:45.254196 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43219
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:45.363421 21471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:45.363731 21471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:45.363806 21471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:45.367985 21471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:45.368088 21471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:45.368184 21471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:43:45.373667 21471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.2.keytab
--principal=kudu/127.2.8.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:45.374785 21471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:45.375844 21471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:45.385536 21477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:45.385586 21479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:45.385581 21476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:45.385756 21471 server_base.cc:1034] running on GCE node
I20260924 10:43:45.386175 21471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:45.386824 21471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:45.388017 21471 hybrid_clock.cc:648] HybridClock initialized: now 1790246625387996 us; error 37 us; skew 500 ppm
Sep 24 10:43:45 dist-test-slave-j38h krb5kdc[21230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246625, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:45.390894 21471 init.cc:377] Logged in from keytab as kudu/127.2.8.2@KRBTEST.COM (short username kudu)
I20260924 10:43:45.391750 21471 webserver.cc:460] Webserver started at http://127.2.8.2:42231/ using document root <none> and password file <none>
I20260924 10:43:45.392318 21471 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:45.392411 21471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:45.392668 21471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:45.394767 21471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "cdf099152fc64ecaa22837301d422530"
format_stamp: "Formatted at 2026-09-24 10:43:45 on dist-test-slave-j38h"
server_key: "ab8415314b6e8b90898d132920beeef5"
server_key_iv: "2cb8829e5017dff1bc9f48d6e16a0bf6"
server_key_version: "encryptionkey@0"
I20260924 10:43:45.395335 21471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "cdf099152fc64ecaa22837301d422530"
format_stamp: "Formatted at 2026-09-24 10:43:45 on dist-test-slave-j38h"
server_key: "ab8415314b6e8b90898d132920beeef5"
server_key_iv: "2cb8829e5017dff1bc9f48d6e16a0bf6"
server_key_version: "encryptionkey@0"
I20260924 10:43:45.399061 21471 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:43:45.401762 21486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:45.403110 21471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:43:45.403242 21471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "cdf099152fc64ecaa22837301d422530"
format_stamp: "Formatted at 2026-09-24 10:43:45 on dist-test-slave-j38h"
server_key: "ab8415314b6e8b90898d132920beeef5"
server_key_iv: "2cb8829e5017dff1bc9f48d6e16a0bf6"
server_key_version: "encryptionkey@0"
I20260924 10:43:45.403353 21471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:45.430696 21471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:45.434131 21471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:45.434343 21471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:45.434957 21471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:45.435798 21471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:45.435869 21471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:45.435941 21471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:45.435994 21471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:45.446105 21471 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:42497
I20260924 10:43:45.446105 21599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:42497 every 8 connection(s)
I20260924 10:43:45.447136 21471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:43:45.450793 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 21471
I20260924 10:43:45.450947 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:43:45.451222 2080 external_mini_cluster.cc:1428] Setting key 81ae3f1b6144a1baa3a739030a94c4df
Sep 24 10:43:45 dist-test-slave-j38h krb5kdc[21230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246625, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.2@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.8.3@KRBTEST.COM; defaulting to no policy
I20260924 10:43:45.461866 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.449064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:56627 (local address 127.2.8.62:43219)
0924 10:43:45.449204 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:45.449208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:45.450055 (+ 847us) server_negotiation.cc:407] Connection header received
0924 10:43:45.451815 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:45.451818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:45.451872 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:45.451951 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:45.453453 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.454070 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.455221 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.455459 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.455567 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.458049 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:45.458057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:45.458061 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:45.458095 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:45.459747 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:45.460267 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:45.460272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:45.460275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:45.460368 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:45.460695 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:45.460700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:45.460703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:45.460960 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:45.461077 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:45.461595 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 10:43:45.461695 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:43:45.462702 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.449449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43219 (local address 127.2.8.2:56627)
0924 10:43:45.449908 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:45.449950 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:45.451567 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:45.452170 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:45.452177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:45.452594 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:45.453316 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:45.453339 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.454196 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.454200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:45.455086 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:45.455095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.455567 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.455621 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:45.455799 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.456406 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:43:45.456423 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:45.457878 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:45.459871 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:45.459876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:45.459888 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:45.460144 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:45.460466 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:45.460468 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:45.460469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:45.460591 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:45.461096 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:45.461101 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:45.461336 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 10:43:45.461502 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 10:43:45.461812 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.3@KRBTEST.COM" created.
I20260924 10:43:45.463985 21600 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43219
I20260924 10:43:45.464231 21600 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:45.464706 21600 heartbeater.cc:511] Master 127.2.8.62:43219 requested a full tablet report, sending...
I20260924 10:43:45.465899 21275 ts_manager.cc:194] Registered new tserver with Master: cdf099152fc64ecaa22837301d422530 (127.2.8.2:42497)
I20260924 10:43:45.466486 21275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.2@KRBTEST.COM'} at 127.2.8.2:56627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
Entry for principal kudu/127.2.8.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab.
I20260924 10:43:45.478417 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43219
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:45.588804 21605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:45.589082 21605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:45.589125 21605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:45.593137 21605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:45.593217 21605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:45.593286 21605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:43:45.598796 21605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.3.keytab
--principal=kudu/127.2.8.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:45.599786 21605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:45.600874 21605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:45.610702 21611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:45.610670 21613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:45.610752 21605 server_base.cc:1034] running on GCE node
W20260924 10:43:45.610679 21610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:45.611403 21605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:45.612186 21605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:45.613404 21605 hybrid_clock.cc:648] HybridClock initialized: now 1790246625613389 us; error 34 us; skew 500 ppm
Sep 24 10:43:45 dist-test-slave-j38h krb5kdc[21230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246625, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:45.616501 21605 init.cc:377] Logged in from keytab as kudu/127.2.8.3@KRBTEST.COM (short username kudu)
I20260924 10:43:45.617632 21605 webserver.cc:460] Webserver started at http://127.2.8.3:33481/ using document root <none> and password file <none>
I20260924 10:43:45.618286 21605 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:45.618392 21605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:45.618654 21605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:45.620940 21605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "4860d220f3414cfb81fdf13a9c18c6f3"
format_stamp: "Formatted at 2026-09-24 10:43:45 on dist-test-slave-j38h"
server_key: "dc91f6ee8f70802d7d93aa6d418c77fd"
server_key_iv: "f1fe6156ae97652fcddeeace06913ecf"
server_key_version: "encryptionkey@0"
I20260924 10:43:45.621680 21605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "4860d220f3414cfb81fdf13a9c18c6f3"
format_stamp: "Formatted at 2026-09-24 10:43:45 on dist-test-slave-j38h"
server_key: "dc91f6ee8f70802d7d93aa6d418c77fd"
server_key_iv: "f1fe6156ae97652fcddeeace06913ecf"
server_key_version: "encryptionkey@0"
I20260924 10:43:45.625921 21605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:43:45.628633 21620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:45.629951 21605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:43:45.630112 21605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "4860d220f3414cfb81fdf13a9c18c6f3"
format_stamp: "Formatted at 2026-09-24 10:43:45 on dist-test-slave-j38h"
server_key: "dc91f6ee8f70802d7d93aa6d418c77fd"
server_key_iv: "f1fe6156ae97652fcddeeace06913ecf"
server_key_version: "encryptionkey@0"
I20260924 10:43:45.630246 21605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:45.651168 21605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:45.654791 21605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:45.655035 21605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:45.655731 21605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:45.656694 21605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:45.656786 21605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:45.656898 21605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:45.656961 21605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:45.666791 21605 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:46547
I20260924 10:43:45.666810 21733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:46547 every 8 connection(s)
I20260924 10:43:45.667763 21605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:43:45.675048 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 21605
I20260924 10:43:45.675169 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:43:45.675472 2080 external_mini_cluster.cc:1428] Setting key f6bbdcc4a55aaa0757b980476ba65dd7
Sep 24 10:43:45 dist-test-slave-j38h krb5kdc[21230](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246625, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.3@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:43:45.683111 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.669654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:46861 (local address 127.2.8.62:43219)
0924 10:43:45.669818 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:45.669821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:45.670690 (+ 869us) server_negotiation.cc:407] Connection header received
0924 10:43:45.672206 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:45.672211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:45.672279 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:45.672418 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:45.673832 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.674442 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.675956 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.676215 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.676302 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.679388 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:45.679392 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:45.679394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:45.679432 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:45.681024 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:45.681557 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:45.681561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:45.681563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:45.681648 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:45.681999 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:45.682004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:45.682006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:45.682198 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:45.682304 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:45.682821 (+ 517us) server_negotiation.cc:299] Negotiation successful
0924 10:43:45.682944 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:43:45.684032 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.669979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43219 (local address 127.2.8.3:46861)
0924 10:43:45.670494 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:45.670543 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:45.671953 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:45.672610 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:45.672618 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:45.673098 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:45.673640 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:45.673655 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.674610 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.674615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:45.675795 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:45.675810 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.676327 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.676389 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:45.676628 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.677398 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:43:45.677415 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:45.679196 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:45.681149 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:45.681156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:45.681178 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:45.681439 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:45.681753 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:43:45.681761 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:43:45.681763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:43:45.681898 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:43:45.682315 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:45.682323 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:45.682582 (+ 259us) client_negotiation.cc:769] Sending connection context
0924 10:43:45.682781 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 10:43:45.683142 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":120,"threads_started":1}
I20260924 10:43:45.685307 21734 heartbeater.cc:347] Connected to a master server at 127.2.8.62:43219
I20260924 10:43:45.685529 21734 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:45.686115 21734 heartbeater.cc:511] Master 127.2.8.62:43219 requested a full tablet report, sending...
I20260924 10:43:45.687201 21275 ts_manager.cc:194] Registered new tserver with Master: 4860d220f3414cfb81fdf13a9c18c6f3 (127.2.8.3:46547)
I20260924 10:43:45.687773 21275 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.8.3@KRBTEST.COM'} at 127.2.8.3:46861
I20260924 10:43:45.690034 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:43:45.699970 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.692190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.2.8.62:43219)
0924 10:43:45.692361 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:45.692365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:45.692471 (+ 106us) server_negotiation.cc:407] Connection header received
0924 10:43:45.692576 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:45.692578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:45.692616 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:45.692681 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:45.693497 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.693996 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.695115 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.695323 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.695400 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.696749 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:45.696752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:45.696754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:45.696776 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:45.698352 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:45.698729 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:45.698732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:45.698733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:45.698803 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:45.699026 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:45.699028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:45.699029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:45.699233 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:45.699333 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:45.699675 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 10:43:45.699768 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 10:43:45.703213 21275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:43:45.706725 21275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:43:45.721105 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.715513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:46547 (local address 127.0.0.1:37832)
0924 10:43:45.716416 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:45.716428 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:45.716521 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:45.717072 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:45.717077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:45.717099 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:45.717262 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:45.717268 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.718366 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.718372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:45.719534 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:45.719543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.720516 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.720520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:45.720627 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.720668 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 10:43:45.720780 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 10:43:45.720898 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":83,"threads_started":1}
I20260924 10:43:45.721122 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.716180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.2.8.3:46547)
0924 10:43:45.716583 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:45.716596 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:45.716619 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:43:45.716695 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:45.716701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:45.716890 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:45.717068 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:45.717426 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.718230 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.719655 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.720388 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.720492 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.720634 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:45.720828 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 10:43:45.720922 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":100,"threads_started":1}
I20260924 10:43:45.721591 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.715309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:41431 (local address 127.0.0.1:45644)
0924 10:43:45.715992 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:45.716043 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:45.716173 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:45.717262 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:45.717265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:45.717280 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:45.717409 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:45.717416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.718783 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.718786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:45.720032 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:45.720042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.721140 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.721146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:45.721260 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.721283 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 10:43:45.721345 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 10:43:45.721405 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260924 10:43:45.721614 21748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.715601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:42497 (local address 127.0.0.1:53074)
0924 10:43:45.716656 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:45.716668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:45.716741 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:45.717402 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:45.717406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:45.717428 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:45.717552 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:45.717562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.719019 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.719022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:45.720149 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:45.720158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.721179 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.721183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:45.721301 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.721323 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 10:43:45.721367 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 10:43:45.721434 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"thread_start_us":87,"threads_started":1}
I20260924 10:43:45.722065 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.715487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.2.8.1:41431)
0924 10:43:45.716757 (+ 1270us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:45.716778 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:45.716803 (+ 25us) server_negotiation.cc:407] Connection header received
0924 10:43:45.716881 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:45.716885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:45.717101 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:45.717278 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:45.717522 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.718639 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.720168 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.720967 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.721552 (+ 585us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.721701 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:45.721755 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 10:43:45.721864 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1166,"thread_start_us":119,"threads_started":1}
I20260924 10:43:45.722416 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.716270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.2.8.2:42497)
0924 10:43:45.717004 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:45.717014 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:45.717034 (+ 20us) server_negotiation.cc:407] Connection header received
0924 10:43:45.717100 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:45.717105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:45.717264 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:45.717702 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:45.717848 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.718872 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.720267 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.721043 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.722009 (+ 966us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.722142 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:45.722186 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 10:43:45.722265 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":95,"threads_started":1}
I20260924 10:43:45.724530 21400 tablet_service.cc:1467] Processing CreateTablet for tablet b30e3d934473428ea4c901a831300b93 (DEFAULT_TABLE table=test-table [id=b9c1d12fe0c94f28a9f1f46e8ea03e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:45.724642 21668 tablet_service.cc:1467] Processing CreateTablet for tablet b30e3d934473428ea4c901a831300b93 (DEFAULT_TABLE table=test-table [id=b9c1d12fe0c94f28a9f1f46e8ea03e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:45.724700 21534 tablet_service.cc:1467] Processing CreateTablet for tablet b30e3d934473428ea4c901a831300b93 (DEFAULT_TABLE table=test-table [id=b9c1d12fe0c94f28a9f1f46e8ea03e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:45.725567 21400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b30e3d934473428ea4c901a831300b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:45.725654 21668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b30e3d934473428ea4c901a831300b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:45.725654 21534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b30e3d934473428ea4c901a831300b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:45.731961 21751 tablet_bootstrap.cc:492] T b30e3d934473428ea4c901a831300b93 P 7e7dcfa6d3b0484cb6e92f1c4c94110f: Bootstrap starting.
I20260924 10:43:45.733402 21752 tablet_bootstrap.cc:492] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3: Bootstrap starting.
I20260924 10:43:45.733402 21753 tablet_bootstrap.cc:492] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530: Bootstrap starting.
I20260924 10:43:45.734294 21751 tablet_bootstrap.cc:654] T b30e3d934473428ea4c901a831300b93 P 7e7dcfa6d3b0484cb6e92f1c4c94110f: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:45.735520 21751 log.cc:824] T b30e3d934473428ea4c901a831300b93 P 7e7dcfa6d3b0484cb6e92f1c4c94110f: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:45.735636 21753 tablet_bootstrap.cc:654] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:45.735630 21752 tablet_bootstrap.cc:654] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:45.736956 21752 log.cc:824] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:45.736944 21753 log.cc:824] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:45.737900 21751 tablet_bootstrap.cc:492] T b30e3d934473428ea4c901a831300b93 P 7e7dcfa6d3b0484cb6e92f1c4c94110f: No bootstrap required, opened a new log
I20260924 10:43:45.738123 21751 ts_tablet_manager.cc:1397] T b30e3d934473428ea4c901a831300b93 P 7e7dcfa6d3b0484cb6e92f1c4c94110f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 10:43:45.739034 21752 tablet_bootstrap.cc:492] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3: No bootstrap required, opened a new log
I20260924 10:43:45.739034 21753 tablet_bootstrap.cc:492] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530: No bootstrap required, opened a new log
I20260924 10:43:45.739219 21753 ts_tablet_manager.cc:1397] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:43:45.739219 21752 ts_tablet_manager.cc:1397] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:43:45.741159 21751 raft_consensus.cc:348] T b30e3d934473428ea4c901a831300b93 P 7e7dcfa6d3b0484cb6e92f1c4c94110f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7dcfa6d3b0484cb6e92f1c4c94110f" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41431 } } peers { permanent_uuid: "4860d220f3414cfb81fdf13a9c18c6f3" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46547 } } peers { permanent_uuid: "cdf099152fc64ecaa22837301d422530" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42497 } }
I20260924 10:43:45.741398 21751 raft_consensus.cc:374] T b30e3d934473428ea4c901a831300b93 P 7e7dcfa6d3b0484cb6e92f1c4c94110f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:45.741468 21751 raft_consensus.cc:729] T b30e3d934473428ea4c901a831300b93 P 7e7dcfa6d3b0484cb6e92f1c4c94110f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7dcfa6d3b0484cb6e92f1c4c94110f, State: Initialized, Role: FOLLOWER
I20260924 10:43:45.742005 21751 consensus_queue.cc:260] T b30e3d934473428ea4c901a831300b93 P 7e7dcfa6d3b0484cb6e92f1c4c94110f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7dcfa6d3b0484cb6e92f1c4c94110f" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41431 } } peers { permanent_uuid: "4860d220f3414cfb81fdf13a9c18c6f3" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46547 } } peers { permanent_uuid: "cdf099152fc64ecaa22837301d422530" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42497 } }
I20260924 10:43:45.742964 21751 ts_tablet_manager.cc:1428] T b30e3d934473428ea4c901a831300b93 P 7e7dcfa6d3b0484cb6e92f1c4c94110f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 10:43:45.743157 21752 raft_consensus.cc:348] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7dcfa6d3b0484cb6e92f1c4c94110f" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41431 } } peers { permanent_uuid: "4860d220f3414cfb81fdf13a9c18c6f3" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46547 } } peers { permanent_uuid: "cdf099152fc64ecaa22837301d422530" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42497 } }
I20260924 10:43:45.743157 21753 raft_consensus.cc:348] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7dcfa6d3b0484cb6e92f1c4c94110f" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41431 } } peers { permanent_uuid: "4860d220f3414cfb81fdf13a9c18c6f3" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46547 } } peers { permanent_uuid: "cdf099152fc64ecaa22837301d422530" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42497 } }
I20260924 10:43:45.743351 21466 heartbeater.cc:503] Master 127.2.8.62:43219 was elected leader, sending a full tablet report...
I20260924 10:43:45.743417 21753 raft_consensus.cc:374] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:45.743496 21752 raft_consensus.cc:374] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:45.743495 21753 raft_consensus.cc:729] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdf099152fc64ecaa22837301d422530, State: Initialized, Role: FOLLOWER
I20260924 10:43:45.743553 21752 raft_consensus.cc:729] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4860d220f3414cfb81fdf13a9c18c6f3, State: Initialized, Role: FOLLOWER
I20260924 10:43:45.743948 21752 consensus_queue.cc:260] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7dcfa6d3b0484cb6e92f1c4c94110f" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41431 } } peers { permanent_uuid: "4860d220f3414cfb81fdf13a9c18c6f3" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46547 } } peers { permanent_uuid: "cdf099152fc64ecaa22837301d422530" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42497 } }
I20260924 10:43:45.743957 21753 consensus_queue.cc:260] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7dcfa6d3b0484cb6e92f1c4c94110f" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41431 } } peers { permanent_uuid: "4860d220f3414cfb81fdf13a9c18c6f3" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46547 } } peers { permanent_uuid: "cdf099152fc64ecaa22837301d422530" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42497 } }
I20260924 10:43:45.744599 21734 heartbeater.cc:503] Master 127.2.8.62:43219 was elected leader, sending a full tablet report...
I20260924 10:43:45.744856 21600 heartbeater.cc:503] Master 127.2.8.62:43219 was elected leader, sending a full tablet report...
I20260924 10:43:45.744899 21753 ts_tablet_manager.cc:1428] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:43:45.745592 21752 ts_tablet_manager.cc:1428] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 10:43:45.796397 21758 raft_consensus.cc:482] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:43:45.796672 21758 raft_consensus.cc:504] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7dcfa6d3b0484cb6e92f1c4c94110f" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41431 } } peers { permanent_uuid: "4860d220f3414cfb81fdf13a9c18c6f3" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46547 } } peers { permanent_uuid: "cdf099152fc64ecaa22837301d422530" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42497 } }
I20260924 10:43:45.797995 21758 leader_election.cc:288] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e7dcfa6d3b0484cb6e92f1c4c94110f (127.2.8.1:41431), cdf099152fc64ecaa22837301d422530 (127.2.8.2:42497)
I20260924 10:43:45.802215 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.798241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:41431 (local address 127.2.8.3:51561)
0924 10:43:45.798409 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:45.798426 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:45.798538 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:45.798833 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:45.798835 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:45.798851 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:45.798993 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:45.799001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.799961 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.799964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:45.801134 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:45.801143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.801761 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.801766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:45.801879 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.801914 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 10:43:45.801987 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 10:43:45.802065 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":32}
I20260924 10:43:45.802246 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.798459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:51561 (local address 127.2.8.1:41431)
0924 10:43:45.798578 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:45.798581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:45.798593 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:43:45.798671 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:45.798674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:45.798725 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:45.798821 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:45.799113 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.799857 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.801258 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.801647 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.801729 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.801780 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:45.802017 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 10:43:45.802094 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 10:43:45.802409 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.798243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:42497 (local address 127.2.8.3:35225)
0924 10:43:45.798824 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:45.798838 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:45.798933 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:45.799186 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:45.799190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:45.799209 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:45.799358 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:45.799364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.800154 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.800156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:45.801130 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:45.801140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.801979 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.801985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:45.802089 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.802110 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 10:43:45.802188 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 10:43:45.802250 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":222,"thread_start_us":139,"threads_started":1}
I20260924 10:43:45.802433 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.798375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:35225 (local address 127.2.8.2:42497)
0924 10:43:45.798508 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:45.798513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:45.798922 (+ 409us) server_negotiation.cc:407] Connection header received
0924 10:43:45.799021 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:45.799026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:45.799084 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:45.799162 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:45.799467 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.800046 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.801250 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.801828 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.801930 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.802012 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:45.802221 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 10:43:45.802301 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 10:43:45.802985 21420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b30e3d934473428ea4c901a831300b93" candidate_uuid: "4860d220f3414cfb81fdf13a9c18c6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7dcfa6d3b0484cb6e92f1c4c94110f" is_pre_election: true
I20260924 10:43:45.803076 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b30e3d934473428ea4c901a831300b93" candidate_uuid: "4860d220f3414cfb81fdf13a9c18c6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdf099152fc64ecaa22837301d422530" is_pre_election: true
I20260924 10:43:45.803313 21420 raft_consensus.cc:2454] T b30e3d934473428ea4c901a831300b93 P 7e7dcfa6d3b0484cb6e92f1c4c94110f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4860d220f3414cfb81fdf13a9c18c6f3 in term 0.
I20260924 10:43:45.803372 21554 raft_consensus.cc:2454] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4860d220f3414cfb81fdf13a9c18c6f3 in term 0.
I20260924 10:43:45.803851 21622 leader_election.cc:302] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4860d220f3414cfb81fdf13a9c18c6f3, 7e7dcfa6d3b0484cb6e92f1c4c94110f, cdf099152fc64ecaa22837301d422530; no voters:
I20260924 10:43:45.804138 21758 raft_consensus.cc:2781] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:43:45.804236 21758 raft_consensus.cc:482] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:43:45.804297 21758 raft_consensus.cc:3037] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:45.805387 21758 raft_consensus.cc:504] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7dcfa6d3b0484cb6e92f1c4c94110f" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41431 } } peers { permanent_uuid: "4860d220f3414cfb81fdf13a9c18c6f3" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46547 } } peers { permanent_uuid: "cdf099152fc64ecaa22837301d422530" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42497 } }
I20260924 10:43:45.805743 21758 leader_election.cc:288] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [CANDIDATE]: Term 1 election: Requested vote from peers 7e7dcfa6d3b0484cb6e92f1c4c94110f (127.2.8.1:41431), cdf099152fc64ecaa22837301d422530 (127.2.8.2:42497)
I20260924 10:43:45.806119 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b30e3d934473428ea4c901a831300b93" candidate_uuid: "4860d220f3414cfb81fdf13a9c18c6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdf099152fc64ecaa22837301d422530"
I20260924 10:43:45.806144 21420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b30e3d934473428ea4c901a831300b93" candidate_uuid: "4860d220f3414cfb81fdf13a9c18c6f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e7dcfa6d3b0484cb6e92f1c4c94110f"
I20260924 10:43:45.806252 21554 raft_consensus.cc:3037] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:45.806252 21420 raft_consensus.cc:3037] T b30e3d934473428ea4c901a831300b93 P 7e7dcfa6d3b0484cb6e92f1c4c94110f [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:45.807559 21420 raft_consensus.cc:2454] T b30e3d934473428ea4c901a831300b93 P 7e7dcfa6d3b0484cb6e92f1c4c94110f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4860d220f3414cfb81fdf13a9c18c6f3 in term 1.
I20260924 10:43:45.807559 21554 raft_consensus.cc:2454] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4860d220f3414cfb81fdf13a9c18c6f3 in term 1.
I20260924 10:43:45.807925 21623 leader_election.cc:302] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4860d220f3414cfb81fdf13a9c18c6f3, 7e7dcfa6d3b0484cb6e92f1c4c94110f, cdf099152fc64ecaa22837301d422530; no voters:
I20260924 10:43:45.808144 21758 raft_consensus.cc:2781] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:45.808346 21758 raft_consensus.cc:686] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [term 1 LEADER]: Becoming Leader. State: Replica: 4860d220f3414cfb81fdf13a9c18c6f3, State: Running, Role: LEADER
I20260924 10:43:45.808694 21758 consensus_queue.cc:237] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7dcfa6d3b0484cb6e92f1c4c94110f" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41431 } } peers { permanent_uuid: "4860d220f3414cfb81fdf13a9c18c6f3" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46547 } } peers { permanent_uuid: "cdf099152fc64ecaa22837301d422530" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42497 } }
I20260924 10:43:45.812222 21274 catalog_manager.cc:5495] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4860d220f3414cfb81fdf13a9c18c6f3 (127.2.8.3). New cstate: current_term: 1 leader_uuid: "4860d220f3414cfb81fdf13a9c18c6f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e7dcfa6d3b0484cb6e92f1c4c94110f" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4860d220f3414cfb81fdf13a9c18c6f3" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 46547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdf099152fc64ecaa22837301d422530" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42497 } health_report { overall_health: UNKNOWN } } }
I20260924 10:43:45.824683 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.821023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.2.8.3:46547)
0924 10:43:45.821158 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:45.821161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:45.821176 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:43:45.821352 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:45.821355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:45.821415 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:45.821507 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:43:45.821707 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.822246 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.823076 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:45.823343 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:45.823429 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:45.823752 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:43:45.824205 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:43:45.824321 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:45.824412 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 10:43:45.824463 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:43:45.830585 21420 raft_consensus.cc:1261] T b30e3d934473428ea4c901a831300b93 P 7e7dcfa6d3b0484cb6e92f1c4c94110f [term 1 FOLLOWER]: Refusing update from remote peer 4860d220f3414cfb81fdf13a9c18c6f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:43:45.830648 21554 raft_consensus.cc:1261] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530 [term 1 FOLLOWER]: Refusing update from remote peer 4860d220f3414cfb81fdf13a9c18c6f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:43:45.831420 21761 consensus_queue.cc:1035] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e7dcfa6d3b0484cb6e92f1c4c94110f" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 41431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:43:45.831728 21758 consensus_queue.cc:1035] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdf099152fc64ecaa22837301d422530" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 42497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:43:45.839903 21765 mvcc.cc:205] Tried to move back new op lower bound from 7332850179394568192 to 7332850179315519488. Current Snapshot: MvccSnapshot[applied={T|T < 7332850179394568192}]
I20260924 10:43:45.840306 21766 mvcc.cc:205] Tried to move back new op lower bound from 7332850179394568192 to 7332850179315519488. Current Snapshot: MvccSnapshot[applied={T|T < 7332850179394568192}]
I20260924 10:43:45.849660 21274 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:43:45.849859 21274 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:43:45.852523 21274 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bead6555fa7348e1b2fd5a9f8ce8f008: Sending DeleteTablet for 3 replicas of tablet b30e3d934473428ea4c901a831300b93
I20260924 10:43:45.853403 21668 tablet_service.cc:1514] Processing DeleteTablet for tablet b30e3d934473428ea4c901a831300b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:45 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:37832
I20260924 10:43:45.853457 21534 tablet_service.cc:1514] Processing DeleteTablet for tablet b30e3d934473428ea4c901a831300b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:45 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:53074
I20260924 10:43:45.853958 21775 tablet_replica.cc:330] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530: stopping tablet replica
I20260924 10:43:45.853977 21400 tablet_service.cc:1514] Processing DeleteTablet for tablet b30e3d934473428ea4c901a831300b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:45 UTC) from {username='kudu', principal='kudu/127.2.8.62@KRBTEST.COM'} at 127.0.0.1:45644
I20260924 10:43:45.854184 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 21337
I20260924 10:43:45.854341 21775 raft_consensus.cc:2229] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:43:45.854696 21775 raft_consensus.cc:2258] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:43:45.855012 21774 tablet_replica.cc:330] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3: stopping tablet replica
I20260924 10:43:45.855352 21774 raft_consensus.cc:2229] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:43:45.855747 21774 raft_consensus.cc:2258] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:43:45.857935 21775 ts_tablet_manager.cc:1905] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:43:45.858656 21774 ts_tablet_manager.cc:1905] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:43:45.860836 21775 ts_tablet_manager.cc:1918] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:43:45.861032 21775 log.cc:1196] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/b30e3d934473428ea4c901a831300b93
I20260924 10:43:45.861449 21775 ts_tablet_manager.cc:1939] T b30e3d934473428ea4c901a831300b93 P cdf099152fc64ecaa22837301d422530: Deleting consensus metadata
I20260924 10:43:45.861490 21774 ts_tablet_manager.cc:1918] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:43:45.861625 21774 log.cc:1196] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/b30e3d934473428ea4c901a831300b93
I20260924 10:43:45.861981 21774 ts_tablet_manager.cc:1939] T b30e3d934473428ea4c901a831300b93 P 4860d220f3414cfb81fdf13a9c18c6f3: Deleting consensus metadata
I20260924 10:43:45.862846 21260 catalog_manager.cc:4841] TS cdf099152fc64ecaa22837301d422530 (127.2.8.2:42497): tablet b30e3d934473428ea4c901a831300b93 (table test-table [id=b9c1d12fe0c94f28a9f1f46e8ea03e90]) successfully deleted
I20260924 10:43:45.863010 21262 catalog_manager.cc:4841] TS 4860d220f3414cfb81fdf13a9c18c6f3 (127.2.8.3:46547): tablet b30e3d934473428ea4c901a831300b93 (table test-table [id=b9c1d12fe0c94f28a9f1f46e8ea03e90]) successfully deleted
W20260924 10:43:45.863368 21261 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.8.1:41431 (error 108)
I20260924 10:43:45.863551 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 21471
I20260924 10:43:45.865916 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:45.865528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:41431 (local address 127.0.0.1:45650)
0924 10:43:45.865706 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:45.865784 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 10:43:45.866154 21261 catalog_manager.cc:4584] TS 7e7dcfa6d3b0484cb6e92f1c4c94110f (127.2.8.1:41431): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b30e3d934473428ea4c901a831300b93: Network error: Client connection negotiation failed: client connection to 127.2.8.1:41431: connect: Connection refused (error 111)
I20260924 10:43:45.870150 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 21605
I20260924 10:43:45.875669 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 21243
2026-09-24T10:43:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2366 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T10:43:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:43:45Z Disabled control of system clock
I20260924 10:43:45.905776 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:39371
--webserver_interface=127.2.8.62
--webserver_port=0
--builtin_ntp_servers=127.2.8.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:39371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:43:46.014832 21780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:46.015093 21780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:46.015128 21780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:46.019052 21780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:43:46.019107 21780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:46.019124 21780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:46.019140 21780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:43:46.019155 21780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:43:46.023932 21780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:39371
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:39371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:46.024966 21780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:46.026280 21780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:46.034557 21786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:46.034560 21785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:46.034560 21788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:46.034924 21780 server_base.cc:1034] running on GCE node
I20260924 10:43:46.035475 21780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:46.036542 21780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:46.037727 21780 hybrid_clock.cc:648] HybridClock initialized: now 1790246626037715 us; error 23 us; skew 500 ppm
I20260924 10:43:46.039623 21780 webserver.cc:460] Webserver started at http://127.2.8.62:34627/ using document root <none> and password file <none>
I20260924 10:43:46.040215 21780 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:46.040310 21780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:46.040549 21780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:46.042659 21780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "37fbc5f482b2462a9042a9810cc7f61b"
format_stamp: "Formatted at 2026-09-24 10:43:46 on dist-test-slave-j38h"
server_key: "ed98b4363f7f1278bd9814831c5b5ffa"
server_key_iv: "b6b7dcb9130f00b0b590b427046f94a7"
server_key_version: "encryptionkey@0"
I20260924 10:43:46.043215 21780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "37fbc5f482b2462a9042a9810cc7f61b"
format_stamp: "Formatted at 2026-09-24 10:43:46 on dist-test-slave-j38h"
server_key: "ed98b4363f7f1278bd9814831c5b5ffa"
server_key_iv: "b6b7dcb9130f00b0b590b427046f94a7"
server_key_version: "encryptionkey@0"
I20260924 10:43:46.046866 21780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 10:43:46.049342 21794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:46.050536 21780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:43:46.050683 21780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "37fbc5f482b2462a9042a9810cc7f61b"
format_stamp: "Formatted at 2026-09-24 10:43:46 on dist-test-slave-j38h"
server_key: "ed98b4363f7f1278bd9814831c5b5ffa"
server_key_iv: "b6b7dcb9130f00b0b590b427046f94a7"
server_key_version: "encryptionkey@0"
I20260924 10:43:46.050797 21780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:46.070191 21780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:46.070977 21780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:46.071192 21780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:46.077801 21780 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:39371
I20260924 10:43:46.077823 21846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:39371 every 8 connection(s)
I20260924 10:43:46.078742 21780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:43:46.081629 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 21780
I20260924 10:43:46.081763 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:43:46.081887 21847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:46.082100 2080 external_mini_cluster.cc:1428] Setting key c7b29e1c1555385297b23ea9367175d0
I20260924 10:43:46.088712 21847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b: Bootstrap starting.
I20260924 10:43:46.091395 21847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:46.092157 21847 log.cc:824] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:46.092319 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.083639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.2.8.62:39371)
0924 10:43:46.084308 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:46.084330 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:46.084368 (+ 38us) server_negotiation.cc:407] Connection header received
0924 10:43:46.085216 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:46.085242 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:46.085682 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:46.086003 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:46.086353 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.087411 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.088665 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.089330 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.089471 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.090415 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:46.090420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:46.090432 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:46.090836 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:46.090892 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:46.090912 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:46.090992 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:46.091182 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 10:43:46.091541 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":134,"threads_started":1}
I20260924 10:43:46.094250 21847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b: No bootstrap required, opened a new log
I20260924 10:43:46.098054 21847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37fbc5f482b2462a9042a9810cc7f61b" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 39371 } }
I20260924 10:43:46.098299 21847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:46.098379 21847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fbc5f482b2462a9042a9810cc7f61b, State: Initialized, Role: FOLLOWER
I20260924 10:43:46.098793 21847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37fbc5f482b2462a9042a9810cc7f61b" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 39371 } }
I20260924 10:43:46.098948 21847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:43:46.099031 21847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:43:46.099149 21847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:46.100076 21847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37fbc5f482b2462a9042a9810cc7f61b" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 39371 } }
I20260924 10:43:46.100415 21847 leader_election.cc:302] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37fbc5f482b2462a9042a9810cc7f61b; no voters:
I20260924 10:43:46.100716 21847 leader_election.cc:288] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:43:46.100852 21852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:46.101117 21852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [term 1 LEADER]: Becoming Leader. State: Replica: 37fbc5f482b2462a9042a9810cc7f61b, State: Running, Role: LEADER
I20260924 10:43:46.101476 21852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37fbc5f482b2462a9042a9810cc7f61b" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 39371 } }
I20260924 10:43:46.102042 21847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:43:46.103142 21853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37fbc5f482b2462a9042a9810cc7f61b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37fbc5f482b2462a9042a9810cc7f61b" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 39371 } } }
I20260924 10:43:46.103193 21854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37fbc5f482b2462a9042a9810cc7f61b. Latest consensus state: current_term: 1 leader_uuid: "37fbc5f482b2462a9042a9810cc7f61b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37fbc5f482b2462a9042a9810cc7f61b" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 39371 } } }
I20260924 10:43:46.103367 21853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:46.103389 21854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:46.104314 21861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:43:46.106972 21861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:43:46.112090 21861 catalog_manager.cc:1347] Generated new cluster ID: ce6890f2f23943bf9e5bb56c5d5d5d68
I20260924 10:43:46.112161 21861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:43:46.129765 21861 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:43:46.131070 21861 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:43:46.143621 21861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b: Generated new TSK 0
I20260924 10:43:46.144382 21861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:43:46.157555 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.1:0
--local_ip_for_outbound_sockets=127.2.8.1
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:39371
--builtin_ntp_servers=127.2.8.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 10:43:46.270890 21871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:46.271139 21871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:46.271176 21871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:46.275063 21871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:46.275110 21871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:46.275161 21871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.1
I20260924 10:43:46.279702 21871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.8.1
--webserver_port=0
--tserver_master_addrs=127.2.8.62:39371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.1
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:46.280720 21871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:46.282088 21871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:46.291414 21879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:46.291442 21876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:46.291442 21877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:46.291712 21871 server_base.cc:1034] running on GCE node
I20260924 10:43:46.292285 21871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:46.293059 21871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:46.294294 21871 hybrid_clock.cc:648] HybridClock initialized: now 1790246626294254 us; error 33 us; skew 500 ppm
I20260924 10:43:46.296272 21871 webserver.cc:460] Webserver started at http://127.2.8.1:35609/ using document root <none> and password file <none>
I20260924 10:43:46.296931 21871 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:46.297058 21871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:46.297303 21871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:46.299425 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/data/instance:
uuid: "b3a2f0c21fae4fb0900ff2999d963ed7"
format_stamp: "Formatted at 2026-09-24 10:43:46 on dist-test-slave-j38h"
server_key: "b62f5e69804b62e3e3d8fbc7e51bfbd0"
server_key_iv: "9793f537e4523e8fbdcec2d34e2e6e56"
server_key_version: "encryptionkey@0"
I20260924 10:43:46.299988 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance:
uuid: "b3a2f0c21fae4fb0900ff2999d963ed7"
format_stamp: "Formatted at 2026-09-24 10:43:46 on dist-test-slave-j38h"
server_key: "b62f5e69804b62e3e3d8fbc7e51bfbd0"
server_key_iv: "9793f537e4523e8fbdcec2d34e2e6e56"
server_key_version: "encryptionkey@0"
I20260924 10:43:46.303867 21871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 10:43:46.306490 21885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:46.307673 21871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:43:46.307806 21871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/wal
uuid: "b3a2f0c21fae4fb0900ff2999d963ed7"
format_stamp: "Formatted at 2026-09-24 10:43:46 on dist-test-slave-j38h"
server_key: "b62f5e69804b62e3e3d8fbc7e51bfbd0"
server_key_iv: "9793f537e4523e8fbdcec2d34e2e6e56"
server_key_version: "encryptionkey@0"
I20260924 10:43:46.307919 21871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:46.327983 21871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:46.328706 21871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:46.328909 21871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:46.329550 21871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:46.330513 21871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:46.330560 21871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:46.330628 21871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:46.330659 21871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:46.339284 21871 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.1:36059
I20260924 10:43:46.339426 21998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.1:36059 every 8 connection(s)
I20260924 10:43:46.340322 21871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/data/info.pb
I20260924 10:43:46.344751 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 21871
I20260924 10:43:46.344897 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-0/wal/instance
I20260924 10:43:46.345255 2080 external_mini_cluster.cc:1428] Setting key 9c057443aa6148c9c9f2d1edcf31d1fa
I20260924 10:43:46.347177 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.2:0
--local_ip_for_outbound_sockets=127.2.8.2
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:39371
--builtin_ntp_servers=127.2.8.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:43:46.349929 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.342267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.1:41401 (local address 127.2.8.62:39371)
0924 10:43:46.342423 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:46.342427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:46.343492 (+ 1065us) server_negotiation.cc:407] Connection header received
0924 10:43:46.344953 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:46.344957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:46.345039 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:46.345143 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:46.346111 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.346666 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.347924 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.348166 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.348282 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.349409 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:46.349412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:46.349415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:46.349428 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:46.349461 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:46.349470 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:46.349547 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:46.349719 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 10:43:46.349785 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 10:43:46.350843 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.342673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:39371 (local address 127.2.8.1:41401)
0924 10:43:46.343325 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:46.343365 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:46.344689 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:46.345303 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:46.345309 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:46.345439 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:46.345952 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:46.345965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.346795 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.346799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:46.347796 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:46.347806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.348268 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.348325 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:46.348495 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.349142 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:43:46.349143 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:46.349185 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:43:46.349249 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:46.349561 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:46.349574 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:46.349584 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 10:43:46.349708 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 10:43:46.350004 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260924 10:43:46.352242 21999 heartbeater.cc:347] Connected to a master server at 127.2.8.62:39371
I20260924 10:43:46.352478 21999 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:46.353040 21999 heartbeater.cc:511] Master 127.2.8.62:39371 requested a full tablet report, sending...
I20260924 10:43:46.354748 21811 ts_manager.cc:194] Registered new tserver with Master: b3a2f0c21fae4fb0900ff2999d963ed7 (127.2.8.1:36059)
I20260924 10:43:46.356514 21811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.8.1:41401
W20260924 10:43:46.460259 22002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:46.460525 22002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:46.460562 22002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:46.464516 22002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:46.464576 22002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:46.464632 22002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.2
I20260924 10:43:46.469345 22002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.8.2
--webserver_port=0
--tserver_master_addrs=127.2.8.62:39371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.2
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:46.470372 22002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:46.471707 22002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:46.481380 22008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:46.481383 22010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:46.481410 22007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:46.481575 22002 server_base.cc:1034] running on GCE node
I20260924 10:43:46.482143 22002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:46.482810 22002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:46.484005 22002 hybrid_clock.cc:648] HybridClock initialized: now 1790246626483994 us; error 35 us; skew 500 ppm
I20260924 10:43:46.485848 22002 webserver.cc:460] Webserver started at http://127.2.8.2:40339/ using document root <none> and password file <none>
I20260924 10:43:46.486438 22002 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:46.486528 22002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:46.486755 22002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:46.489063 22002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/data/instance:
uuid: "cdab1fb329004c9b93158be19c13fb23"
format_stamp: "Formatted at 2026-09-24 10:43:46 on dist-test-slave-j38h"
server_key: "d4f37948e138aa8a806fd0fa137fffcb"
server_key_iv: "7ff8732a99f50d79515043d4f23ab6f2"
server_key_version: "encryptionkey@0"
I20260924 10:43:46.489636 22002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance:
uuid: "cdab1fb329004c9b93158be19c13fb23"
format_stamp: "Formatted at 2026-09-24 10:43:46 on dist-test-slave-j38h"
server_key: "d4f37948e138aa8a806fd0fa137fffcb"
server_key_iv: "7ff8732a99f50d79515043d4f23ab6f2"
server_key_version: "encryptionkey@0"
I20260924 10:43:46.493294 22002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 10:43:46.495754 22016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:46.497021 22002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:43:46.497164 22002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/wal
uuid: "cdab1fb329004c9b93158be19c13fb23"
format_stamp: "Formatted at 2026-09-24 10:43:46 on dist-test-slave-j38h"
server_key: "d4f37948e138aa8a806fd0fa137fffcb"
server_key_iv: "7ff8732a99f50d79515043d4f23ab6f2"
server_key_version: "encryptionkey@0"
I20260924 10:43:46.497278 22002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:46.514513 22002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:46.515220 22002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:46.515419 22002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:46.516008 22002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:46.516932 22002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:46.517050 22002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:46.517162 22002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:46.517213 22002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:46.525801 22002 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.2:43223
I20260924 10:43:46.525823 22129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.2:43223 every 8 connection(s)
I20260924 10:43:46.526751 22002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/data/info.pb
I20260924 10:43:46.533685 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 22002
I20260924 10:43:46.533821 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/wal/instance
I20260924 10:43:46.534103 2080 external_mini_cluster.cc:1428] Setting key fed95362cb1280a0aa45fad03955d5e1
I20260924 10:43:46.536250 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.8.3:0
--local_ip_for_outbound_sockets=127.2.8.3
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:39371
--builtin_ntp_servers=127.2.8.20:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 10:43:46.536466 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.528593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.2:45451 (local address 127.2.8.62:39371)
0924 10:43:46.528740 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:46.528743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:46.529783 (+ 1040us) server_negotiation.cc:407] Connection header received
0924 10:43:46.531138 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:46.531141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:46.531186 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:46.531263 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:46.532130 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.532700 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.534388 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.534675 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.534761 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.535924 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:46.535927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:46.535928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:46.535947 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:46.535967 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:46.535975 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:46.536034 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:46.536221 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 10:43:46.536275 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:43:46.537649 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.529018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:39371 (local address 127.2.8.2:45451)
0924 10:43:46.529657 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:46.529697 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:46.530933 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:46.531426 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:46.531433 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:46.531518 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:46.532006 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:46.532019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.532820 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.532824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:46.534267 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:46.534277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.534758 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.534810 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:46.534989 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.535631 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:43:46.535633 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:46.535688 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:43:46.535767 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:46.536060 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:46.536072 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:46.536081 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 10:43:46.536221 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 10:43:46.536544 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":104,"threads_started":1}
I20260924 10:43:46.538780 22130 heartbeater.cc:347] Connected to a master server at 127.2.8.62:39371
I20260924 10:43:46.539021 22130 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:46.539660 22130 heartbeater.cc:511] Master 127.2.8.62:39371 requested a full tablet report, sending...
I20260924 10:43:46.540781 21811 ts_manager.cc:194] Registered new tserver with Master: cdab1fb329004c9b93158be19c13fb23 (127.2.8.2:43223)
I20260924 10:43:46.541460 21811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.8.2:45451
W20260924 10:43:46.651404 22133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:46.651711 22133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:46.651782 22133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:46.655805 22133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:46.655900 22133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:46.655997 22133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.8.3
I20260924 10:43:46.660826 22133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.8.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.8.3
--webserver_port=0
--tserver_master_addrs=127.2.8.62:39371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.8.3
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:46.661989 22133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:46.663395 22133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:46.673303 22139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:46.673303 22138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:46.673297 22141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:46.673663 22133 server_base.cc:1034] running on GCE node
I20260924 10:43:46.674131 22133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:46.674855 22133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:46.676072 22133 hybrid_clock.cc:648] HybridClock initialized: now 1790246626676034 us; error 50 us; skew 500 ppm
I20260924 10:43:46.687014 22133 webserver.cc:460] Webserver started at http://127.2.8.3:39067/ using document root <none> and password file <none>
I20260924 10:43:46.687707 22133 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:46.687811 22133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:46.688060 22133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:46.690297 22133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/data/instance:
uuid: "51dbd65bb9444810b5f64d601e8466ad"
format_stamp: "Formatted at 2026-09-24 10:43:46 on dist-test-slave-j38h"
server_key: "8cb4aa137e3b10b4d4a310148d059041"
server_key_iv: "ccb5396c32a9e5129f182608153b2931"
server_key_version: "encryptionkey@0"
I20260924 10:43:46.690874 22133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance:
uuid: "51dbd65bb9444810b5f64d601e8466ad"
format_stamp: "Formatted at 2026-09-24 10:43:46 on dist-test-slave-j38h"
server_key: "8cb4aa137e3b10b4d4a310148d059041"
server_key_iv: "ccb5396c32a9e5129f182608153b2931"
server_key_version: "encryptionkey@0"
I20260924 10:43:46.694702 22133 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 10:43:46.697244 22147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:46.698539 22133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:43:46.698639 22133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/wal
uuid: "51dbd65bb9444810b5f64d601e8466ad"
format_stamp: "Formatted at 2026-09-24 10:43:46 on dist-test-slave-j38h"
server_key: "8cb4aa137e3b10b4d4a310148d059041"
server_key_iv: "ccb5396c32a9e5129f182608153b2931"
server_key_version: "encryptionkey@0"
I20260924 10:43:46.698729 22133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:46.739120 22133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:46.739965 22133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:46.740141 22133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:46.740694 22133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 10:43:46.741664 22133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 10:43:46.741711 22133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:46.741750 22133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 10:43:46.741766 22133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:46.751534 22133 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.3:45281
I20260924 10:43:46.751576 22260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.3:45281 every 8 connection(s)
I20260924 10:43:46.752574 22133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/data/info.pb
I20260924 10:43:46.754004 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 22133
I20260924 10:43:46.754153 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/wal/instance
I20260924 10:43:46.754460 2080 external_mini_cluster.cc:1428] Setting key a69e803954113a9efe893a3ea72fba6b
I20260924 10:43:46.762367 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.754653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:37973 (local address 127.2.8.62:39371)
0924 10:43:46.754808 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:46.754812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:46.756285 (+ 1473us) server_negotiation.cc:407] Connection header received
0924 10:43:46.757817 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:46.757819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:46.757864 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:46.757929 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:46.758829 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.759336 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.760488 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.760698 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.760772 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.761874 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:46.761877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:46.761879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:46.761890 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:46.761908 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:46.761916 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:46.761979 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:46.762200 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 10:43:46.762251 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 10:43:46.763391 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.755267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:39371 (local address 127.2.8.3:37973)
0924 10:43:46.756120 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:46.756174 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:46.757628 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:46.758099 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:46.758105 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:46.758189 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:43:46.758710 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:46.758722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.759441 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.759443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:46.760373 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:46.760382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.760781 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.760823 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:46.761023 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.761635 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 10:43:46.761636 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:43:46.761675 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 10:43:46.761737 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:43:46.762028 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:43:46.762038 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:43:46.762043 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 10:43:46.762194 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 10:43:46.762453 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":149,"threads_started":1}
I20260924 10:43:46.764529 22261 heartbeater.cc:347] Connected to a master server at 127.2.8.62:39371
I20260924 10:43:46.764793 22261 heartbeater.cc:464] Registering TS with master...
I20260924 10:43:46.765467 22261 heartbeater.cc:511] Master 127.2.8.62:39371 requested a full tablet report, sending...
I20260924 10:43:46.766582 21811 ts_manager.cc:194] Registered new tserver with Master: 51dbd65bb9444810b5f64d601e8466ad (127.2.8.3:45281)
I20260924 10:43:46.767309 21811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.8.3:37973
I20260924 10:43:46.768626 2080 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 10:43:46.774024 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.770695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55124 (local address 127.2.8.62:39371)
0924 10:43:46.770812 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:46.770815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:46.771025 (+ 210us) server_negotiation.cc:407] Connection header received
0924 10:43:46.771113 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:46.771115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:46.771166 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:46.771231 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:46.771462 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.771933 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.772692 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.772888 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.772963 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.773706 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:46.773709 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:46.773711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:46.773722 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:46.773738 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:46.773746 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:46.773803 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:46.773879 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 10:43:46.773921 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:43:46.776630 21811 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 10:43:46.779644 21811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 10:43:46.794723 22273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.788466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:36059 (local address 127.0.0.1:59750)
0924 10:43:46.789189 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:46.789204 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:46.789312 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:46.790219 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:46.790222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:46.790256 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:46.790408 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:46.790414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.791566 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.791571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:46.793018 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:46.793027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.794038 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.794043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:46.794175 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.794232 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 10:43:46.794366 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 10:43:46.794514 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":28,"spinlock_wait_cycles":896,"thread_start_us":85,"threads_started":1}
I20260924 10:43:46.794955 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.788472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.2.8.2:43223)
0924 10:43:46.788854 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:46.788870 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:46.789465 (+ 595us) server_negotiation.cc:407] Connection header received
0924 10:43:46.789562 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:46.789568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:46.789826 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:46.790006 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:46.790643 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.791985 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.793405 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.794193 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.794343 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.794546 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:46.794607 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 10:43:46.794713 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":138,"threads_started":1}
I20260924 10:43:46.794747 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.788316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:43223 (local address 127.0.0.1:41014)
0924 10:43:46.789067 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:46.789122 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:46.789293 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:46.790212 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:46.790218 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:46.790280 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:46.790447 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:46.790455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.792122 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.792125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:46.793271 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:46.793278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.794327 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.794331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:46.794419 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.794460 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 10:43:46.794520 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 10:43:46.794587 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":40,"spinlock_wait_cycles":72576,"thread_start_us":89,"threads_started":1}
I20260924 10:43:46.795356 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.789611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.3:45281 (local address 127.0.0.1:40352)
0924 10:43:46.789900 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:46.789919 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:46.790015 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:46.790577 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:46.790579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:46.790594 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:46.790673 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:46.790680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.792178 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.792183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:46.793500 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:46.793512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.794761 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.794765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:46.794864 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.794884 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 10:43:46.794951 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 10:43:46.795210 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":52,"thread_start_us":101,"threads_started":1}
I20260924 10:43:46.795557 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.789204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59750 (local address 127.2.8.1:36059)
0924 10:43:46.789775 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:46.789786 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:46.789809 (+ 23us) server_negotiation.cc:407] Connection header received
0924 10:43:46.789884 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:46.789889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:46.790072 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:46.790563 (+ 491us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:46.790698 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.791451 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.793144 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.793891 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.793995 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.794175 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:46.795115 (+ 940us) server_negotiation.cc:299] Negotiation successful
0924 10:43:46.795353 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":134,"threads_started":1}
I20260924 10:43:46.795846 22276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.789236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.2.8.3:45281)
0924 10:43:46.789577 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:46.789590 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:46.790127 (+ 537us) server_negotiation.cc:407] Connection header received
0924 10:43:46.790219 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:46.790225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:46.790416 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:46.790608 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:46.790798 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.792023 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.793644 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.794420 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.794528 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.794699 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:46.795584 (+ 885us) server_negotiation.cc:299] Negotiation successful
0924 10:43:46.795695 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":162,"threads_started":1}
I20260924 10:43:46.798393 22064 tablet_service.cc:1467] Processing CreateTablet for tablet 56a2d3f8bb4344d39f8a17647707a67c (DEFAULT_TABLE table=test-table [id=1061358a752f4204a93c7487cbc7ca4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:46.798542 22195 tablet_service.cc:1467] Processing CreateTablet for tablet 56a2d3f8bb4344d39f8a17647707a67c (DEFAULT_TABLE table=test-table [id=1061358a752f4204a93c7487cbc7ca4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:46.798542 21933 tablet_service.cc:1467] Processing CreateTablet for tablet 56a2d3f8bb4344d39f8a17647707a67c (DEFAULT_TABLE table=test-table [id=1061358a752f4204a93c7487cbc7ca4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 10:43:46.799423 22064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56a2d3f8bb4344d39f8a17647707a67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:46.799629 21933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56a2d3f8bb4344d39f8a17647707a67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:46.799629 22195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56a2d3f8bb4344d39f8a17647707a67c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:46.806524 22278 tablet_bootstrap.cc:492] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23: Bootstrap starting.
I20260924 10:43:46.808287 22280 tablet_bootstrap.cc:492] T 56a2d3f8bb4344d39f8a17647707a67c P b3a2f0c21fae4fb0900ff2999d963ed7: Bootstrap starting.
I20260924 10:43:46.808353 22279 tablet_bootstrap.cc:492] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad: Bootstrap starting.
I20260924 10:43:46.808866 22278 tablet_bootstrap.cc:654] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:46.809944 22278 log.cc:824] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:46.810998 22280 tablet_bootstrap.cc:654] T 56a2d3f8bb4344d39f8a17647707a67c P b3a2f0c21fae4fb0900ff2999d963ed7: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:46.810998 22279 tablet_bootstrap.cc:654] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:46.811980 22279 log.cc:824] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:46.811980 22280 log.cc:824] T 56a2d3f8bb4344d39f8a17647707a67c P b3a2f0c21fae4fb0900ff2999d963ed7: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:46.812408 22278 tablet_bootstrap.cc:492] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23: No bootstrap required, opened a new log
I20260924 10:43:46.812623 22278 ts_tablet_manager.cc:1397] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:43:46.814136 22279 tablet_bootstrap.cc:492] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad: No bootstrap required, opened a new log
I20260924 10:43:46.814136 22280 tablet_bootstrap.cc:492] T 56a2d3f8bb4344d39f8a17647707a67c P b3a2f0c21fae4fb0900ff2999d963ed7: No bootstrap required, opened a new log
I20260924 10:43:46.814329 22280 ts_tablet_manager.cc:1397] T 56a2d3f8bb4344d39f8a17647707a67c P b3a2f0c21fae4fb0900ff2999d963ed7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:43:46.814329 22279 ts_tablet_manager.cc:1397] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 10:43:46.816061 22278 raft_consensus.cc:348] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdab1fb329004c9b93158be19c13fb23" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 43223 } } peers { permanent_uuid: "b3a2f0c21fae4fb0900ff2999d963ed7" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 36059 } } peers { permanent_uuid: "51dbd65bb9444810b5f64d601e8466ad" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45281 } }
I20260924 10:43:46.816304 22278 raft_consensus.cc:374] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:46.816375 22278 raft_consensus.cc:729] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdab1fb329004c9b93158be19c13fb23, State: Initialized, Role: FOLLOWER
I20260924 10:43:46.816861 22278 consensus_queue.cc:260] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdab1fb329004c9b93158be19c13fb23" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 43223 } } peers { permanent_uuid: "b3a2f0c21fae4fb0900ff2999d963ed7" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 36059 } } peers { permanent_uuid: "51dbd65bb9444810b5f64d601e8466ad" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45281 } }
I20260924 10:43:46.817536 22130 heartbeater.cc:503] Master 127.2.8.62:39371 was elected leader, sending a full tablet report...
I20260924 10:43:46.817914 22278 ts_tablet_manager.cc:1428] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 10:43:46.819052 22280 raft_consensus.cc:348] T 56a2d3f8bb4344d39f8a17647707a67c P b3a2f0c21fae4fb0900ff2999d963ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdab1fb329004c9b93158be19c13fb23" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 43223 } } peers { permanent_uuid: "b3a2f0c21fae4fb0900ff2999d963ed7" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 36059 } } peers { permanent_uuid: "51dbd65bb9444810b5f64d601e8466ad" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45281 } }
I20260924 10:43:46.819052 22279 raft_consensus.cc:348] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdab1fb329004c9b93158be19c13fb23" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 43223 } } peers { permanent_uuid: "b3a2f0c21fae4fb0900ff2999d963ed7" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 36059 } } peers { permanent_uuid: "51dbd65bb9444810b5f64d601e8466ad" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45281 } }
I20260924 10:43:46.819331 22279 raft_consensus.cc:374] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:46.819418 22280 raft_consensus.cc:374] T 56a2d3f8bb4344d39f8a17647707a67c P b3a2f0c21fae4fb0900ff2999d963ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:46.819434 22279 raft_consensus.cc:729] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51dbd65bb9444810b5f64d601e8466ad, State: Initialized, Role: FOLLOWER
I20260924 10:43:46.819542 22280 raft_consensus.cc:729] T 56a2d3f8bb4344d39f8a17647707a67c P b3a2f0c21fae4fb0900ff2999d963ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a2f0c21fae4fb0900ff2999d963ed7, State: Initialized, Role: FOLLOWER
I20260924 10:43:46.819937 22279 consensus_queue.cc:260] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdab1fb329004c9b93158be19c13fb23" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 43223 } } peers { permanent_uuid: "b3a2f0c21fae4fb0900ff2999d963ed7" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 36059 } } peers { permanent_uuid: "51dbd65bb9444810b5f64d601e8466ad" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45281 } }
I20260924 10:43:46.819962 22280 consensus_queue.cc:260] T 56a2d3f8bb4344d39f8a17647707a67c P b3a2f0c21fae4fb0900ff2999d963ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdab1fb329004c9b93158be19c13fb23" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 43223 } } peers { permanent_uuid: "b3a2f0c21fae4fb0900ff2999d963ed7" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 36059 } } peers { permanent_uuid: "51dbd65bb9444810b5f64d601e8466ad" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45281 } }
I20260924 10:43:46.821050 21999 heartbeater.cc:503] Master 127.2.8.62:39371 was elected leader, sending a full tablet report...
I20260924 10:43:46.821075 22279 ts_tablet_manager.cc:1428] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260924 10:43:46.821075 22280 ts_tablet_manager.cc:1428] T 56a2d3f8bb4344d39f8a17647707a67c P b3a2f0c21fae4fb0900ff2999d963ed7: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260924 10:43:46.821420 22261 heartbeater.cc:503] Master 127.2.8.62:39371 was elected leader, sending a full tablet report...
W20260924 10:43:46.841622 22000 tablet.cc:2369] T 56a2d3f8bb4344d39f8a17647707a67c P b3a2f0c21fae4fb0900ff2999d963ed7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:43:46.913254 22286 raft_consensus.cc:482] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 10:43:46.913516 22286 raft_consensus.cc:504] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdab1fb329004c9b93158be19c13fb23" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 43223 } } peers { permanent_uuid: "b3a2f0c21fae4fb0900ff2999d963ed7" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 36059 } } peers { permanent_uuid: "51dbd65bb9444810b5f64d601e8466ad" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45281 } }
I20260924 10:43:46.914739 22286 leader_election.cc:288] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdab1fb329004c9b93158be19c13fb23 (127.2.8.2:43223), b3a2f0c21fae4fb0900ff2999d963ed7 (127.2.8.1:36059)
I20260924 10:43:46.919749 22275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.915137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:49073 (local address 127.2.8.1:36059)
0924 10:43:46.915266 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:46.915270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:46.915622 (+ 352us) server_negotiation.cc:407] Connection header received
0924 10:43:46.915785 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:46.915788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:46.915833 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:46.915894 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:46.916166 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.916919 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.918494 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.919083 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.919208 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.919268 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:46.919544 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 10:43:46.919619 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 10:43:46.919749 22274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.915137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.8.3:43181 (local address 127.2.8.2:43223)
0924 10:43:46.915287 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:46.915294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:46.915323 (+ 29us) server_negotiation.cc:407] Connection header received
0924 10:43:46.915489 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:46.915491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:46.915537 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:46.915605 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 10:43:46.915972 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.916774 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.918494 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.919085 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.919207 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.919267 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:46.919544 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 10:43:46.919619 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 10:43:46.919754 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.915038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.2:43223 (local address 127.2.8.3:43181)
0924 10:43:46.915170 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:46.915187 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:46.915290 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:46.915680 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:46.915685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:46.915703 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:46.915833 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:46.915839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.916906 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.916909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:46.918351 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:46.918356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.919242 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.919246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:46.919417 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.919442 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 10:43:46.919512 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 10:43:46.919593 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":72}
I20260924 10:43:46.919762 22287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.915038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:36059 (local address 127.2.8.3:49073)
0924 10:43:46.915533 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:46.915543 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 10:43:46.915665 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:43:46.915926 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:43:46.915930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:43:46.915949 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 10:43:46.916055 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:46.916059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.917040 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.917044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:46.918343 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:43:46.918354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.919242 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.919246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:43:46.919374 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.919404 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 10:43:46.919499 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 10:43:46.919592 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":213,"thread_start_us":112,"threads_started":1}
I20260924 10:43:46.920583 22084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a2d3f8bb4344d39f8a17647707a67c" candidate_uuid: "51dbd65bb9444810b5f64d601e8466ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdab1fb329004c9b93158be19c13fb23" is_pre_election: true
I20260924 10:43:46.920795 21953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a2d3f8bb4344d39f8a17647707a67c" candidate_uuid: "51dbd65bb9444810b5f64d601e8466ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a2f0c21fae4fb0900ff2999d963ed7" is_pre_election: true
I20260924 10:43:46.920930 22084 raft_consensus.cc:2454] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51dbd65bb9444810b5f64d601e8466ad in term 0.
I20260924 10:43:46.921131 21953 raft_consensus.cc:2454] T 56a2d3f8bb4344d39f8a17647707a67c P b3a2f0c21fae4fb0900ff2999d963ed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51dbd65bb9444810b5f64d601e8466ad in term 0.
I20260924 10:43:46.921528 22149 leader_election.cc:302] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51dbd65bb9444810b5f64d601e8466ad, cdab1fb329004c9b93158be19c13fb23; no voters:
I20260924 10:43:46.921844 22286 raft_consensus.cc:2781] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 10:43:46.921926 22286 raft_consensus.cc:482] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 10:43:46.921978 22286 raft_consensus.cc:3037] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:46.922993 22286 raft_consensus.cc:504] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdab1fb329004c9b93158be19c13fb23" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 43223 } } peers { permanent_uuid: "b3a2f0c21fae4fb0900ff2999d963ed7" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 36059 } } peers { permanent_uuid: "51dbd65bb9444810b5f64d601e8466ad" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45281 } }
I20260924 10:43:46.923360 22286 leader_election.cc:288] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [CANDIDATE]: Term 1 election: Requested vote from peers cdab1fb329004c9b93158be19c13fb23 (127.2.8.2:43223), b3a2f0c21fae4fb0900ff2999d963ed7 (127.2.8.1:36059)
I20260924 10:43:46.923709 22084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a2d3f8bb4344d39f8a17647707a67c" candidate_uuid: "51dbd65bb9444810b5f64d601e8466ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdab1fb329004c9b93158be19c13fb23"
I20260924 10:43:46.923738 21953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a2d3f8bb4344d39f8a17647707a67c" candidate_uuid: "51dbd65bb9444810b5f64d601e8466ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a2f0c21fae4fb0900ff2999d963ed7"
I20260924 10:43:46.923849 22084 raft_consensus.cc:3037] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:46.923857 21953 raft_consensus.cc:3037] T 56a2d3f8bb4344d39f8a17647707a67c P b3a2f0c21fae4fb0900ff2999d963ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:46.924863 21953 raft_consensus.cc:2454] T 56a2d3f8bb4344d39f8a17647707a67c P b3a2f0c21fae4fb0900ff2999d963ed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51dbd65bb9444810b5f64d601e8466ad in term 1.
I20260924 10:43:46.924904 22084 raft_consensus.cc:2454] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51dbd65bb9444810b5f64d601e8466ad in term 1.
I20260924 10:43:46.925207 22151 leader_election.cc:302] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51dbd65bb9444810b5f64d601e8466ad, b3a2f0c21fae4fb0900ff2999d963ed7; no voters:
I20260924 10:43:46.925372 22286 raft_consensus.cc:2781] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:46.925593 22286 raft_consensus.cc:686] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [term 1 LEADER]: Becoming Leader. State: Replica: 51dbd65bb9444810b5f64d601e8466ad, State: Running, Role: LEADER
I20260924 10:43:46.925923 22286 consensus_queue.cc:237] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdab1fb329004c9b93158be19c13fb23" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 43223 } } peers { permanent_uuid: "b3a2f0c21fae4fb0900ff2999d963ed7" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 36059 } } peers { permanent_uuid: "51dbd65bb9444810b5f64d601e8466ad" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45281 } }
I20260924 10:43:46.929297 21810 catalog_manager.cc:5495] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 51dbd65bb9444810b5f64d601e8466ad (127.2.8.3). New cstate: current_term: 1 leader_uuid: "51dbd65bb9444810b5f64d601e8466ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdab1fb329004c9b93158be19c13fb23" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 43223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3a2f0c21fae4fb0900ff2999d963ed7" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 36059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51dbd65bb9444810b5f64d601e8466ad" member_type: VOTER last_known_addr { host: "127.2.8.3" port: 45281 } health_report { overall_health: HEALTHY } } }
I20260924 10:43:46.948704 22276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.945091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.2.8.3:45281)
0924 10:43:46.945231 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:46.945236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:46.945251 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:43:46.945437 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:46.945446 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:46.945501 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:46.945617 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:43:46.945874 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.946399 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.947264 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:46.947493 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:46.947603 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:46.947912 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:43:46.948318 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:43:46.948449 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:46.948488 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 10:43:46.948538 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 10:43:46.954425 21953 raft_consensus.cc:1261] T 56a2d3f8bb4344d39f8a17647707a67c P b3a2f0c21fae4fb0900ff2999d963ed7 [term 1 FOLLOWER]: Refusing update from remote peer 51dbd65bb9444810b5f64d601e8466ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:43:46.954423 22084 raft_consensus.cc:1261] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23 [term 1 FOLLOWER]: Refusing update from remote peer 51dbd65bb9444810b5f64d601e8466ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:43:46.955113 22286 consensus_queue.cc:1035] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3a2f0c21fae4fb0900ff2999d963ed7" member_type: VOTER last_known_addr { host: "127.2.8.1" port: 36059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:43:46.955313 22288 consensus_queue.cc:1035] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdab1fb329004c9b93158be19c13fb23" member_type: VOTER last_known_addr { host: "127.2.8.2" port: 43223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:43:46.962993 22292 mvcc.cc:205] Tried to move back new op lower bound from 7332850183997771776 to 7332850183891013632. Current Snapshot: MvccSnapshot[applied={T|T < 7332850183997771776}]
I20260924 10:43:46.971666 21811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:43:46.971845 21811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 10:43:46.974239 21811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37fbc5f482b2462a9042a9810cc7f61b: Sending DeleteTablet for 3 replicas of tablet 56a2d3f8bb4344d39f8a17647707a67c
I20260924 10:43:46.975106 22195 tablet_service.cc:1514] Processing DeleteTablet for tablet 56a2d3f8bb4344d39f8a17647707a67c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:46 UTC) from {username='slave'} at 127.0.0.1:40352
I20260924 10:43:46.975093 21933 tablet_service.cc:1514] Processing DeleteTablet for tablet 56a2d3f8bb4344d39f8a17647707a67c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:46 UTC) from {username='slave'} at 127.0.0.1:59750
I20260924 10:43:46.975633 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 21871
I20260924 10:43:46.975844 22302 tablet_replica.cc:330] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad: stopping tablet replica
I20260924 10:43:46.975826 22064 tablet_service.cc:1514] Processing DeleteTablet for tablet 56a2d3f8bb4344d39f8a17647707a67c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 10:43:46 UTC) from {username='slave'} at 127.0.0.1:41014
I20260924 10:43:46.976197 22302 raft_consensus.cc:2229] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [term 1 LEADER]: Raft consensus shutting down.
I20260924 10:43:46.976585 22302 raft_consensus.cc:2258] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:43:46.977413 22303 tablet_replica.cc:330] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23: stopping tablet replica
I20260924 10:43:46.977751 22303 raft_consensus.cc:2229] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 10:43:46.978107 22303 raft_consensus.cc:2258] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 10:43:46.978292 22302 ts_tablet_manager.cc:1905] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:43:46.980437 22303 ts_tablet_manager.cc:1905] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 10:43:46.980897 22302 ts_tablet_manager.cc:1918] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:43:46.981024 22302 log.cc:1196] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-2/wal/wals/56a2d3f8bb4344d39f8a17647707a67c
I20260924 10:43:46.981297 22302 ts_tablet_manager.cc:1939] T 56a2d3f8bb4344d39f8a17647707a67c P 51dbd65bb9444810b5f64d601e8466ad: Deleting consensus metadata
I20260924 10:43:46.982723 21796 catalog_manager.cc:4841] TS 51dbd65bb9444810b5f64d601e8466ad (127.2.8.3:45281): tablet 56a2d3f8bb4344d39f8a17647707a67c (table test-table [id=1061358a752f4204a93c7487cbc7ca4d]) successfully deleted
I20260924 10:43:46.983424 22303 ts_tablet_manager.cc:1918] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 10:43:46.983520 22303 log.cc:1196] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23: Deleting WAL directory at /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790246374820021-2080-0/minicluster-data/ts-1/wal/wals/56a2d3f8bb4344d39f8a17647707a67c
I20260924 10:43:46.983831 22303 ts_tablet_manager.cc:1939] T 56a2d3f8bb4344d39f8a17647707a67c P cdab1fb329004c9b93158be19c13fb23: Deleting consensus metadata
I20260924 10:43:46.985153 21796 catalog_manager.cc:4841] TS cdab1fb329004c9b93158be19c13fb23 (127.2.8.2:43223): tablet 56a2d3f8bb4344d39f8a17647707a67c (table test-table [id=1061358a752f4204a93c7487cbc7ca4d]) successfully deleted
W20260924 10:43:46.986094 21798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.8.1:36059 (error 108)
I20260924 10:43:46.986241 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 22002
I20260924 10:43:46.987244 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:46.986935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.1:36059 (local address 127.0.0.1:59756)
0924 10:43:46.987101 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 10:43:46.987174 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.1:36059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260924 10:43:46.987449 21798 catalog_manager.cc:4584] TS b3a2f0c21fae4fb0900ff2999d963ed7 (127.2.8.1:36059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56a2d3f8bb4344d39f8a17647707a67c: Network error: Client connection negotiation failed: client connection to 127.2.8.1:36059: connect: Connection refused (error 111)
I20260924 10:43:46.993448 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 22133
I20260924 10:43:46.998725 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 21780
2026-09-24T10:43:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1121 ms)
[----------] 6 tests from EncryptionPolicyTest (10735 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:43:47 dist-test-slave-j38h krb5kdc[22307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:43:47 dist-test-slave-j38h krb5kdc[22307](info): set up 2 sockets
Sep 24 10:43:47 dist-test-slave-j38h krb5kdc[22307](info): commencing operation
krb5kdc: starting...
W20260924 10:43:48.121258 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:43:48 dist-test-slave-j38h krb5kdc[22307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:43:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:43:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:43:48.243772 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:48.353533 22320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:48.353844 22320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:48.353928 22320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:48.358019 22320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:43:48.358115 22320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:48.358155 22320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:48.358217 22320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:43:48.358255 22320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:43:48.363641 22320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33799
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:48.364717 22320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:48.365801 22320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:48.374310 22325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:48.374421 22320 server_base.cc:1034] running on GCE node
W20260924 10:43:48.374343 22326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:48.374282 22328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:48.375147 22320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:48.376250 22320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:48.377439 22320 hybrid_clock.cc:648] HybridClock initialized: now 1790246628377418 us; error 27 us; skew 500 ppm
Sep 24 10:43:48 dist-test-slave-j38h krb5kdc[22307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:48.380119 22320 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:43:48.380954 22320 webserver.cc:460] Webserver started at http://127.0.0.1:46393/ using document root <none> and password file <none>
I20260924 10:43:48.381549 22320 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:48.381639 22320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:48.381862 22320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:48.383816 22320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "59590cd6803946169d2dfa4c7f998719"
format_stamp: "Formatted at 2026-09-24 10:43:48 on dist-test-slave-j38h"
server_key: "14053ad278c9320b76284a48224cb266"
server_key_iv: "b703d2faf99321fda30d880bfcf039c5"
server_key_version: "encryptionkey@0"
I20260924 10:43:48.384372 22320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "59590cd6803946169d2dfa4c7f998719"
format_stamp: "Formatted at 2026-09-24 10:43:48 on dist-test-slave-j38h"
server_key: "14053ad278c9320b76284a48224cb266"
server_key_iv: "b703d2faf99321fda30d880bfcf039c5"
server_key_version: "encryptionkey@0"
I20260924 10:43:48.388128 22320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:43:48.390637 22335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:48.391896 22320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:43:48.392033 22320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "59590cd6803946169d2dfa4c7f998719"
format_stamp: "Formatted at 2026-09-24 10:43:48 on dist-test-slave-j38h"
server_key: "14053ad278c9320b76284a48224cb266"
server_key_iv: "b703d2faf99321fda30d880bfcf039c5"
server_key_version: "encryptionkey@0"
I20260924 10:43:48.392143 22320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:48.401911 22320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:48.405319 22320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:48.405532 22320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:48.412456 22320 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33799
I20260924 10:43:48.412492 22387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33799 every 8 connection(s)
I20260924 10:43:48.413399 22320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:43:48.416258 22388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:48.419878 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 22320
I20260924 10:43:48.420059 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:43:48.420470 2080 external_mini_cluster.cc:1428] Setting key 3e2f10f852e318215c0260620866984c
I20260924 10:43:48.423022 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719: Bootstrap starting.
I20260924 10:43:48.425477 22388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:48.426290 22388 log.cc:824] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:48.428416 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719: No bootstrap required, opened a new log
Sep 24 10:43:48 dist-test-slave-j38h krb5kdc[22307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:43:48.432696 22388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59590cd6803946169d2dfa4c7f998719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33799 } }
I20260924 10:43:48.432948 22388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:48.433068 22388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59590cd6803946169d2dfa4c7f998719, State: Initialized, Role: FOLLOWER
I20260924 10:43:48.433637 22388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59590cd6803946169d2dfa4c7f998719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33799 } }
I20260924 10:43:48.433844 22388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:43:48.433967 22388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:43:48.434115 22388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:48.435462 22388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59590cd6803946169d2dfa4c7f998719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33799 } }
I20260924 10:43:48.435880 22388 leader_election.cc:302] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59590cd6803946169d2dfa4c7f998719; no voters:
I20260924 10:43:48.436255 22388 leader_election.cc:288] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:43:48.436398 22393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:48.436618 22393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [term 1 LEADER]: Becoming Leader. State: Replica: 59590cd6803946169d2dfa4c7f998719, State: Running, Role: LEADER
I20260924 10:43:48.436717 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:48.422045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48443 (local address 127.0.0.1:33799)
0924 10:43:48.422523 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:48.422538 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:48.422572 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:43:48.423254 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:48.423272 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:48.423614 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:48.423951 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:48.424735 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:48.425855 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:48.426770 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:48.427236 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:48.427400 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:48.430538 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:48.430543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:48.430556 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:48.430583 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:48.432868 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:48.433547 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:48.433554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:48.433560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:48.433730 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:48.434121 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:48.434128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:48.434133 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:48.434814 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:48.435077 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:48.435320 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 10:43:48.435597 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":179,"threads_started":1}
I20260924 10:43:48.436964 22393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59590cd6803946169d2dfa4c7f998719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33799 } }
I20260924 10:43:48.437705 22388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:43:48.439280 22394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59590cd6803946169d2dfa4c7f998719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59590cd6803946169d2dfa4c7f998719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33799 } } }
I20260924 10:43:48.439399 22394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:48.440436 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:43:48.440423 22395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59590cd6803946169d2dfa4c7f998719. Latest consensus state: current_term: 1 leader_uuid: "59590cd6803946169d2dfa4c7f998719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59590cd6803946169d2dfa4c7f998719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33799 } } }
I20260924 10:43:48.440682 22395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719 [sys.catalog]: This master's current role is: LEADER
W20260924 10:43:48.443599 22409 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:43:48.443681 22409 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:43:48.443750 22403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:43:48.444417 22403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:43:48.450057 22403 catalog_manager.cc:1347] Generated new cluster ID: 419eb5f4452742049a863e214d0d56f1
I20260924 10:43:48.450126 22403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:43:48.454728 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:48.442808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35311 (local address 127.0.0.1:33799)
0924 10:43:48.442938 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:48.442944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:48.445294 (+ 2350us) server_negotiation.cc:407] Connection header received
0924 10:43:48.445496 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:48.445499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:48.445546 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:48.445636 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:48.446499 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:48.447243 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:48.448543 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:48.448865 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:48.449023 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:48.450745 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:48.450749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:48.450751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:48.450776 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:48.452861 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:48.453410 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:48.453413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:48.453415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:48.453499 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:48.453817 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:48.453820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:48.453822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:48.454017 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:48.454138 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:48.454485 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 10:43:48.454576 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:43:48.471748 22403 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:43:48.472877 22403 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:43:48.484634 22403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59590cd6803946169d2dfa4c7f998719: Generated new TSK 0
I20260924 10:43:48.485379 22403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:43:48.498353 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 22320
W20260924 10:43:48.503028 22305 connection.cc:535] client connection to 127.0.0.1:33799 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33799): Connection reset by peer (error 104)
2026-09-24T10:43:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1498 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:43:48 dist-test-slave-j38h krb5kdc[22419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:43:48 dist-test-slave-j38h krb5kdc[22419](info): set up 2 sockets
Sep 24 10:43:48 dist-test-slave-j38h krb5kdc[22419](info): commencing operation
krb5kdc: starting...
W20260924 10:43:49.533252 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:43:49 dist-test-slave-j38h krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:43:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:43:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:43:49.656915 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38199
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:49.765676 22432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:49.766013 22432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:49.766095 22432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:49.770200 22432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:43:49.770300 22432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:49.770349 22432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:49.770385 22432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:43:49.770416 22432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:43:49.775597 22432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38199
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:49.776678 22432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:49.777735 22432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:49.786531 22440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:49.786484 22438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:49.786636 22432 server_base.cc:1034] running on GCE node
W20260924 10:43:49.786477 22437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:49.787292 22432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:49.788395 22432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:49.789590 22432 hybrid_clock.cc:648] HybridClock initialized: now 1790246629789581 us; error 29 us; skew 500 ppm
Sep 24 10:43:49 dist-test-slave-j38h krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:49.792272 22432 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:43:49.793241 22432 webserver.cc:460] Webserver started at http://127.0.0.1:40231/ using document root <none> and password file <none>
I20260924 10:43:49.793790 22432 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:49.793883 22432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:49.794117 22432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:49.796152 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "05bc41292aae4112ad4815e7c30ca839"
format_stamp: "Formatted at 2026-09-24 10:43:49 on dist-test-slave-j38h"
server_key: "c1ee081913ad612ad0a18fe3984d42c7"
server_key_iv: "393aaed50d61c59b4a0b5fd28629aada"
server_key_version: "encryptionkey@0"
I20260924 10:43:49.796722 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "05bc41292aae4112ad4815e7c30ca839"
format_stamp: "Formatted at 2026-09-24 10:43:49 on dist-test-slave-j38h"
server_key: "c1ee081913ad612ad0a18fe3984d42c7"
server_key_iv: "393aaed50d61c59b4a0b5fd28629aada"
server_key_version: "encryptionkey@0"
I20260924 10:43:49.800436 22432 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:43:49.802904 22447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:49.804142 22432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:43:49.804281 22432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "05bc41292aae4112ad4815e7c30ca839"
format_stamp: "Formatted at 2026-09-24 10:43:49 on dist-test-slave-j38h"
server_key: "c1ee081913ad612ad0a18fe3984d42c7"
server_key_iv: "393aaed50d61c59b4a0b5fd28629aada"
server_key_version: "encryptionkey@0"
I20260924 10:43:49.804394 22432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:49.843969 22432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:49.847033 22432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:49.847429 22432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:49.854753 22432 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38199
I20260924 10:43:49.854776 22499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38199 every 8 connection(s)
I20260924 10:43:49.855741 22432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:43:49.858705 22500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:49.863473 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 22432
I20260924 10:43:49.863593 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:43:49.863890 2080 external_mini_cluster.cc:1428] Setting key ebc4223339874b00fa8ba5c9b26768ed
I20260924 10:43:49.864506 22500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839: Bootstrap starting.
I20260924 10:43:49.867183 22500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:49.868065 22500 log.cc:824] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:49.870152 22500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839: No bootstrap required, opened a new log
I20260924 10:43:49.873625 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:49.865382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46893 (local address 127.0.0.1:38199)
0924 10:43:49.865847 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:49.865874 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:49.865917 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:43:49.866717 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:49.866740 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:49.867187 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:49.867577 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:49.868029 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:49.869398 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:49.870293 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:49.870778 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:49.870959 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:49.871904 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:49.871909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:49.871920 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:49.872303 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:49.872358 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:49.872379 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:49.872526 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:49.872703 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 10:43:49.872844 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":121,"threads_started":1}
I20260924 10:43:49.874101 22500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc41292aae4112ad4815e7c30ca839" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38199 } }
I20260924 10:43:49.874387 22500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:49.874445 22500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05bc41292aae4112ad4815e7c30ca839, State: Initialized, Role: FOLLOWER
I20260924 10:43:49.874897 22500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc41292aae4112ad4815e7c30ca839" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38199 } }
I20260924 10:43:49.875062 22500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:43:49.875154 22500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:43:49.875283 22500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:49.876484 22500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc41292aae4112ad4815e7c30ca839" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38199 } }
I20260924 10:43:49.876852 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:43:49.876870 22500 leader_election.cc:302] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05bc41292aae4112ad4815e7c30ca839; no voters:
I20260924 10:43:49.877218 22500 leader_election.cc:288] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:43:49.877323 22505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:49.877579 22505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [term 1 LEADER]: Becoming Leader. State: Replica: 05bc41292aae4112ad4815e7c30ca839, State: Running, Role: LEADER
I20260924 10:43:49.877907 22505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc41292aae4112ad4815e7c30ca839" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38199 } }
I20260924 10:43:49.878615 22500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:43:49.879726 22507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05bc41292aae4112ad4815e7c30ca839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc41292aae4112ad4815e7c30ca839" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38199 } } }
I20260924 10:43:49.879863 22507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:49.880290 22506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05bc41292aae4112ad4815e7c30ca839. Latest consensus state: current_term: 1 leader_uuid: "05bc41292aae4112ad4815e7c30ca839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05bc41292aae4112ad4815e7c30ca839" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38199 } } }
I20260924 10:43:49.880451 22506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839 [sys.catalog]: This master's current role is: LEADER
W20260924 10:43:49.883785 22525 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:43:49.883863 22525 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:43:49.883980 22519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:43:49.884279 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:49.879426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44111 (local address 127.0.0.1:38199)
0924 10:43:49.879545 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:49.879549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:49.879776 (+ 227us) server_negotiation.cc:407] Connection header received
0924 10:43:49.879968 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:49.879975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:49.880061 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:49.880119 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:49.880566 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:49.881361 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:49.882236 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:49.882575 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:49.882679 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:49.883658 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:49.883661 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:49.883664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:49.883676 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:49.883709 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:49.883719 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:49.883815 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:49.884049 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 10:43:49.884084 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:43:49.885354 22519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:43:49.890753 22519 catalog_manager.cc:1347] Generated new cluster ID: 8e48193b6c864a23b2f1eba0dc2ad6aa
I20260924 10:43:49.890817 22519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:43:49.899351 22519 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:43:49.900660 22519 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:43:49.914738 22519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05bc41292aae4112ad4815e7c30ca839: Generated new TSK 0
I20260924 10:43:49.915596 22519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:43:49.921630 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 22432
W20260924 10:43:49.927001 22417 connection.cc:535] client connection to 127.0.0.1:38199 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38199): Connection reset by peer (error 104)
2026-09-24T10:43:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:43:49 dist-test-slave-j38h krb5kdc[22531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:43:49 dist-test-slave-j38h krb5kdc[22531](info): set up 2 sockets
Sep 24 10:43:49 dist-test-slave-j38h krb5kdc[22531](info): commencing operation
krb5kdc: starting...
W20260924 10:43:50.965202 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:43:51 dist-test-slave-j38h krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:43:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:43:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:43:51.087926 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:51.198032 22544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:51.198382 22544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:51.198428 22544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:51.202996 22544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:43:51.203068 22544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:51.203090 22544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:51.203109 22544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:43:51.203186 22544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:43:51.209075 22544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:51.210253 22544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:51.211386 22544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:51.220366 22552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:51.220381 22550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:51.220366 22549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:51.220503 22544 server_base.cc:1034] running on GCE node
I20260924 10:43:51.221266 22544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:51.222420 22544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:51.223624 22544 hybrid_clock.cc:648] HybridClock initialized: now 1790246631223611 us; error 31 us; skew 500 ppm
Sep 24 10:43:51 dist-test-slave-j38h krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:51.236869 22544 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:43:51.237942 22544 webserver.cc:460] Webserver started at http://127.0.0.1:46873/ using document root <none> and password file <none>
I20260924 10:43:51.238556 22544 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:51.238652 22544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:51.238889 22544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:51.241055 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "c0b56682ccad4111a4e9679d772faece"
format_stamp: "Formatted at 2026-09-24 10:43:51 on dist-test-slave-j38h"
server_key: "0547a84f45123d3e324dd531bba6b345"
server_key_iv: "8abb128652282b7b7d16c27c89e9e339"
server_key_version: "encryptionkey@0"
I20260924 10:43:51.241649 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "c0b56682ccad4111a4e9679d772faece"
format_stamp: "Formatted at 2026-09-24 10:43:51 on dist-test-slave-j38h"
server_key: "0547a84f45123d3e324dd531bba6b345"
server_key_iv: "8abb128652282b7b7d16c27c89e9e339"
server_key_version: "encryptionkey@0"
I20260924 10:43:51.245437 22544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 10:43:51.247766 22559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:51.249051 22544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:43:51.249179 22544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "c0b56682ccad4111a4e9679d772faece"
format_stamp: "Formatted at 2026-09-24 10:43:51 on dist-test-slave-j38h"
server_key: "0547a84f45123d3e324dd531bba6b345"
server_key_iv: "8abb128652282b7b7d16c27c89e9e339"
server_key_version: "encryptionkey@0"
I20260924 10:43:51.249289 22544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:51.254000 22544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:51.258500 22544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:51.258693 22544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:51.265722 22544 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38833
I20260924 10:43:51.265741 22611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38833 every 8 connection(s)
I20260924 10:43:51.266644 22544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:43:51.269382 22612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:51.273802 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 22544
I20260924 10:43:51.273913 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:43:51.274204 2080 external_mini_cluster.cc:1428] Setting key 2f6d82656f3817141867ff1b918c996f
I20260924 10:43:51.274940 22612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece: Bootstrap starting.
I20260924 10:43:51.277751 22612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:51.278609 22612 log.cc:824] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:51.280580 22612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece: No bootstrap required, opened a new log
I20260924 10:43:51.280598 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:51.275623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34593 (local address 127.0.0.1:38833)
0924 10:43:51.276055 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:51.276072 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:51.276122 (+ 50us) server_negotiation.cc:407] Connection header received
0924 10:43:51.276960 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:51.277017 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:51.277384 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:51.277761 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:51.279407 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:51.279411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:51.279423 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:51.279434 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:51.279491 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:51.279509 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:51.279591 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:51.279723 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 10:43:51.279890 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":118,"threads_started":1}
I20260924 10:43:51.283464 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:43:51.284667 22612 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b56682ccad4111a4e9679d772faece" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38833 } }
I20260924 10:43:51.284940 22612 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:51.285027 22612 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0b56682ccad4111a4e9679d772faece, State: Initialized, Role: FOLLOWER
I20260924 10:43:51.285521 22612 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b56682ccad4111a4e9679d772faece" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38833 } }
I20260924 10:43:51.285699 22612 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:43:51.285795 22612 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:43:51.285900 22612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:51.286792 22612 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b56682ccad4111a4e9679d772faece" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38833 } }
I20260924 10:43:51.287132 22612 leader_election.cc:302] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0b56682ccad4111a4e9679d772faece; no voters:
I20260924 10:43:51.287420 22612 leader_election.cc:288] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:43:51.287559 22622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:51.287763 22622 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [term 1 LEADER]: Becoming Leader. State: Replica: c0b56682ccad4111a4e9679d772faece, State: Running, Role: LEADER
I20260924 10:43:51.288091 22622 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b56682ccad4111a4e9679d772faece" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38833 } }
I20260924 10:43:51.288602 22612 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:43:51.289673 22624 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0b56682ccad4111a4e9679d772faece. Latest consensus state: current_term: 1 leader_uuid: "c0b56682ccad4111a4e9679d772faece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b56682ccad4111a4e9679d772faece" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38833 } } }
I20260924 10:43:51.289788 22624 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:51.290113 22623 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0b56682ccad4111a4e9679d772faece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0b56682ccad4111a4e9679d772faece" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38833 } } }
I20260924 10:43:51.290261 22623 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:51.290470 22631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:43:51.292188 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:51.285470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37895 (local address 127.0.0.1:38833)
0924 10:43:51.285579 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:51.285583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:51.289244 (+ 3661us) server_negotiation.cc:407] Connection header received
0924 10:43:51.289434 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:51.289441 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:51.289485 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:51.289544 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:51.291483 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:51.291490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:51.291496 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:51.291508 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:51.291537 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:51.291550 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:51.291625 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:51.292025 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 10:43:51.292068 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:43:51.293689 22631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:43:51.299016 22631 catalog_manager.cc:1347] Generated new cluster ID: c37d3d28497044448e15428fc93f7966
I20260924 10:43:51.299093 22631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:43:51.334481 22631 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:43:51.334877 22631 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:43:51.344993 22631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0b56682ccad4111a4e9679d772faece: Generated new TSK 0
I20260924 10:43:51.345866 22631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:43:51.358430 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 22544
2026-09-24T10:43:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:43:51 dist-test-slave-j38h krb5kdc[22643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:43:51 dist-test-slave-j38h krb5kdc[22643](info): set up 2 sockets
Sep 24 10:43:51 dist-test-slave-j38h krb5kdc[22643](info): commencing operation
krb5kdc: starting...
W20260924 10:43:52.401216 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:43:52 dist-test-slave-j38h krb5kdc[22643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:43:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:43:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:43:52.532670 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45635
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:52.643379 22656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:52.643746 22656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:52.643841 22656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:52.648212 22656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:43:52.648312 22656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:52.648352 22656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:52.648384 22656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:43:52.648416 22656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:43:52.653785 22656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:52.654948 22656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:52.655994 22656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:52.664460 22662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:52.664499 22661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:52.664740 22656 server_base.cc:1034] running on GCE node
W20260924 10:43:52.664495 22664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:52.665562 22656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:52.666765 22656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:52.667946 22656 hybrid_clock.cc:648] HybridClock initialized: now 1790246632667925 us; error 31 us; skew 500 ppm
Sep 24 10:43:52 dist-test-slave-j38h krb5kdc[22643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:52.671092 22656 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:43:52.672055 22656 webserver.cc:460] Webserver started at http://127.0.0.1:33135/ using document root <none> and password file <none>
I20260924 10:43:52.672632 22656 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:52.672727 22656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:52.672971 22656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:52.675153 22656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "d1c65b4c29834516bcfca55a60f68e7a"
format_stamp: "Formatted at 2026-09-24 10:43:52 on dist-test-slave-j38h"
server_key: "cfbd47f8c711825ace938a0273d434ca"
server_key_iv: "e89e45fa91956a4880e37d982a539617"
server_key_version: "encryptionkey@0"
I20260924 10:43:52.675757 22656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "d1c65b4c29834516bcfca55a60f68e7a"
format_stamp: "Formatted at 2026-09-24 10:43:52 on dist-test-slave-j38h"
server_key: "cfbd47f8c711825ace938a0273d434ca"
server_key_iv: "e89e45fa91956a4880e37d982a539617"
server_key_version: "encryptionkey@0"
I20260924 10:43:52.679517 22656 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:43:52.682127 22671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:52.683441 22656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:43:52.683600 22656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "d1c65b4c29834516bcfca55a60f68e7a"
format_stamp: "Formatted at 2026-09-24 10:43:52 on dist-test-slave-j38h"
server_key: "cfbd47f8c711825ace938a0273d434ca"
server_key_iv: "e89e45fa91956a4880e37d982a539617"
server_key_version: "encryptionkey@0"
I20260924 10:43:52.683727 22656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:52.714660 22656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:52.718001 22656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:52.718230 22656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:52.725620 22656 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45635
I20260924 10:43:52.725643 22723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45635 every 8 connection(s)
I20260924 10:43:52.726557 22656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:43:52.729146 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 22656
I20260924 10:43:52.729290 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:43:52.729370 22724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:52.729642 2080 external_mini_cluster.cc:1428] Setting key e5976dd2ed3ba870e4b9a02859fe1ee0
I20260924 10:43:52.736505 22724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a: Bootstrap starting.
I20260924 10:43:52.739097 22724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a: Neither blocks nor log segments found. Creating new log.
Sep 24 10:43:52 dist-test-slave-j38h krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:43:52.739989 22724 log.cc:824] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:52.742141 22724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a: No bootstrap required, opened a new log
I20260924 10:43:52.744751 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:52.731047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45585 (local address 127.0.0.1:45635)
0924 10:43:52.731473 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:52.731491 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:52.731536 (+ 45us) server_negotiation.cc:407] Connection header received
0924 10:43:52.732712 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:52.732759 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:52.733219 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:52.733585 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:52.734252 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:52.735763 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:52.736712 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:52.737387 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:52.737546 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:52.740237 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:52.740242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:52.740256 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:52.740282 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:52.742054 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:52.742533 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:52.742537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:52.742542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:52.742674 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:52.742974 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:52.742976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:52.742977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:52.743328 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:52.743463 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:52.743756 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 10:43:52.744037 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":129,"threads_started":1}
I20260924 10:43:52.746124 22724 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c65b4c29834516bcfca55a60f68e7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45635 } }
I20260924 10:43:52.746390 22724 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:52.746449 22724 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1c65b4c29834516bcfca55a60f68e7a, State: Initialized, Role: FOLLOWER
I20260924 10:43:52.746939 22724 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c65b4c29834516bcfca55a60f68e7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45635 } }
I20260924 10:43:52.747108 22724 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:43:52.747179 22724 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:43:52.747303 22724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:52.747615 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:43:52.748263 22724 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c65b4c29834516bcfca55a60f68e7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45635 } }
I20260924 10:43:52.748606 22724 leader_election.cc:302] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1c65b4c29834516bcfca55a60f68e7a; no voters:
I20260924 10:43:52.748941 22724 leader_election.cc:288] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:43:52.749107 22732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:52.749294 22732 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [term 1 LEADER]: Becoming Leader. State: Replica: d1c65b4c29834516bcfca55a60f68e7a, State: Running, Role: LEADER
I20260924 10:43:52.749571 22732 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c65b4c29834516bcfca55a60f68e7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45635 } }
I20260924 10:43:52.750174 22724 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:43:52.750694 22732 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1c65b4c29834516bcfca55a60f68e7a. Latest consensus state: current_term: 1 leader_uuid: "d1c65b4c29834516bcfca55a60f68e7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c65b4c29834516bcfca55a60f68e7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45635 } } }
I20260924 10:43:52.750811 22732 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:52.751094 22735 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1c65b4c29834516bcfca55a60f68e7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1c65b4c29834516bcfca55a60f68e7a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45635 } } }
I20260924 10:43:52.751184 22735 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:52.752472 22739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:43:52.755460 22739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:43:52.759433 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:52.749982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35923 (local address 127.0.0.1:45635)
0924 10:43:52.750105 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:52.750109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:52.750122 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:43:52.750216 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:52.750219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:52.750264 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:52.750337 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:52.752677 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:52.753361 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:52.754177 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:52.754516 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:52.754627 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:52.756015 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:52.756019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:52.756021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:52.756045 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:52.757664 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:52.758144 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:52.758148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:52.758150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:52.758231 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:52.758525 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:52.758528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:52.758530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:52.758727 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:52.758822 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:52.759179 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 10:43:52.759303 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 10:43:52.761642 22739 catalog_manager.cc:1347] Generated new cluster ID: e0c1912b648043c7a9c729b1faf0b45b
I20260924 10:43:52.761715 22739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:43:52.773720 22739 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:43:52.774636 22739 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:43:52.791150 22739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1c65b4c29834516bcfca55a60f68e7a: Generated new TSK 0
I20260924 10:43:52.791926 22739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:43:52.796875 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 22656
2026-09-24T10:43:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:43:52 dist-test-slave-j38h krb5kdc[22755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:43:52 dist-test-slave-j38h krb5kdc[22755](info): set up 2 sockets
Sep 24 10:43:52 dist-test-slave-j38h krb5kdc[22755](info): commencing operation
krb5kdc: starting...
W20260924 10:43:53.833340 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:43:53 dist-test-slave-j38h krb5kdc[22755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:43:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:43:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:43:53.954468 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:54.062399 22768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:54.062738 22768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:54.062817 22768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:54.066862 22768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:43:54.066962 22768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:54.067003 22768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:54.067040 22768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:43:54.067075 22768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:43:54.072305 22768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:54.073448 22768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:54.074474 22768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:54.082788 22774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:54.082866 22768 server_base.cc:1034] running on GCE node
W20260924 10:43:54.082904 22773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:54.082904 22776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:54.083606 22768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:54.084661 22768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:54.085863 22768 hybrid_clock.cc:648] HybridClock initialized: now 1790246634085833 us; error 59 us; skew 500 ppm
Sep 24 10:43:54 dist-test-slave-j38h krb5kdc[22755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:54.088532 22768 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:43:54.089504 22768 webserver.cc:460] Webserver started at http://127.0.0.1:40429/ using document root <none> and password file <none>
I20260924 10:43:54.090063 22768 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:54.090155 22768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:54.090385 22768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:54.092348 22768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "6545afa9b4bf4d35909fd0637a18a76a"
format_stamp: "Formatted at 2026-09-24 10:43:54 on dist-test-slave-j38h"
server_key: "28b666325899dedf12b9352bf0c89286"
server_key_iv: "89dad66c19aeeaa407b3beb5c5bd2efd"
server_key_version: "encryptionkey@0"
I20260924 10:43:54.092902 22768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "6545afa9b4bf4d35909fd0637a18a76a"
format_stamp: "Formatted at 2026-09-24 10:43:54 on dist-test-slave-j38h"
server_key: "28b666325899dedf12b9352bf0c89286"
server_key_iv: "89dad66c19aeeaa407b3beb5c5bd2efd"
server_key_version: "encryptionkey@0"
I20260924 10:43:54.096470 22768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:43:54.098882 22783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:54.100092 22768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:43:54.100230 22768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "6545afa9b4bf4d35909fd0637a18a76a"
format_stamp: "Formatted at 2026-09-24 10:43:54 on dist-test-slave-j38h"
server_key: "28b666325899dedf12b9352bf0c89286"
server_key_iv: "89dad66c19aeeaa407b3beb5c5bd2efd"
server_key_version: "encryptionkey@0"
I20260924 10:43:54.100339 22768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:54.127427 22768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:54.130527 22768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:54.130735 22768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:54.138180 22768 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34547
I20260924 10:43:54.138204 22835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34547 every 8 connection(s)
I20260924 10:43:54.139164 22768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:43:54.140673 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 22768
I20260924 10:43:54.140826 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:43:54.141187 2080 external_mini_cluster.cc:1428] Setting key 029c4c1872b3f4f538931f01dae2b8ac
I20260924 10:43:54.142282 22836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:54.147739 22836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a: Bootstrap starting.
I20260924 10:43:54.149966 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:54.142607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45219 (local address 127.0.0.1:34547)
0924 10:43:54.143048 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:54.143062 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:54.143096 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:43:54.143743 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:54.143760 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:54.144088 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:54.144414 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:54.144812 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:54.145841 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:54.146746 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:54.147233 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:54.147395 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:54.148292 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:54.148297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:54.148311 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:54.148689 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:54.148740 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:54.148763 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:54.148874 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:54.149040 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 10:43:54.149247 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":147,"threads_started":1}
I20260924 10:43:54.150044 22836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:54.150743 22836 log.cc:824] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:54.152671 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:43:54.152815 22836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a: No bootstrap required, opened a new log
I20260924 10:43:54.156919 22836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6545afa9b4bf4d35909fd0637a18a76a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } }
I20260924 10:43:54.157243 22836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:54.157322 22836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6545afa9b4bf4d35909fd0637a18a76a, State: Initialized, Role: FOLLOWER
I20260924 10:43:54.157860 22836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6545afa9b4bf4d35909fd0637a18a76a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } }
I20260924 10:43:54.158032 22836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:43:54.158113 22836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:43:54.158214 22836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:54.159085 22839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:54.154755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44581 (local address 127.0.0.1:34547)
0924 10:43:54.154884 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:54.154889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:54.155061 (+ 172us) server_negotiation.cc:407] Connection header received
0924 10:43:54.155200 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:54.155202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:54.155241 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:54.155305 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:54.155669 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:54.156178 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:54.157295 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:54.157651 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:54.157788 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:54.158659 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:54.158662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:54.158665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:54.158676 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:54.158698 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:54.158708 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:54.158771 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:54.158887 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 10:43:54.158934 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:43:54.159502 22836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6545afa9b4bf4d35909fd0637a18a76a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } }
I20260924 10:43:54.159806 22836 leader_election.cc:302] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6545afa9b4bf4d35909fd0637a18a76a; no voters:
I20260924 10:43:54.160123 22836 leader_election.cc:288] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:43:54.160218 22846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:54.160492 22846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [term 1 LEADER]: Becoming Leader. State: Replica: 6545afa9b4bf4d35909fd0637a18a76a, State: Running, Role: LEADER
I20260924 10:43:54.160830 22846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6545afa9b4bf4d35909fd0637a18a76a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } }
I20260924 10:43:54.161389 22836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:43:54.162536 22847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6545afa9b4bf4d35909fd0637a18a76a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6545afa9b4bf4d35909fd0637a18a76a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } } }
I20260924 10:43:54.162663 22847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:54.163048 22855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:43:54.163450 22848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6545afa9b4bf4d35909fd0637a18a76a. Latest consensus state: current_term: 1 leader_uuid: "6545afa9b4bf4d35909fd0637a18a76a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6545afa9b4bf4d35909fd0637a18a76a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } } }
I20260924 10:43:54.163553 22848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:54.166034 22855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:43:54.171562 22855 catalog_manager.cc:1347] Generated new cluster ID: c67d32a7908e4371884d3e9b53467a3b
I20260924 10:43:54.171626 22855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:43:54.183528 22855 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:43:54.184468 22855 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:43:54.188508 22855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6545afa9b4bf4d35909fd0637a18a76a: Generated new TSK 0
I20260924 10:43:54.189216 22855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:43:54.194250 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 22768
2026-09-24T10:43:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:43:54 dist-test-slave-j38h krb5kdc[22867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:43:54 dist-test-slave-j38h krb5kdc[22867](info): set up 2 sockets
Sep 24 10:43:54 dist-test-slave-j38h krb5kdc[22867](info): commencing operation
krb5kdc: starting...
W20260924 10:43:55.241433 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:43:55 dist-test-slave-j38h krb5kdc[22867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:43:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:43:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:43:55.362370 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44511
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:55.473518 22880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:55.473793 22880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:55.473873 22880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:55.478004 22880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:43:55.478065 22880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:55.478084 22880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:55.478142 22880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:43:55.478166 22880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:43:55.483573 22880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:55.484709 22880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:55.485833 22880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:55.495141 22885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:55.495138 22886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:55.495196 22888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:55.495402 22880 server_base.cc:1034] running on GCE node
I20260924 10:43:55.496148 22880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:55.497403 22880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:55.498612 22880 hybrid_clock.cc:648] HybridClock initialized: now 1790246635498586 us; error 55 us; skew 500 ppm
Sep 24 10:43:55 dist-test-slave-j38h krb5kdc[22867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:55.501726 22880 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:43:55.502605 22880 webserver.cc:460] Webserver started at http://127.0.0.1:38197/ using document root <none> and password file <none>
I20260924 10:43:55.503137 22880 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:55.503199 22880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:55.503381 22880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:55.505610 22880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "e547aa30e8eb4de59ef9626bdfd2d400"
format_stamp: "Formatted at 2026-09-24 10:43:55 on dist-test-slave-j38h"
server_key: "aa53d880a60d203d598029963debbfb6"
server_key_iv: "5db29d7fe5655f87ad6b85b59c60cef0"
server_key_version: "encryptionkey@0"
I20260924 10:43:55.506139 22880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "e547aa30e8eb4de59ef9626bdfd2d400"
format_stamp: "Formatted at 2026-09-24 10:43:55 on dist-test-slave-j38h"
server_key: "aa53d880a60d203d598029963debbfb6"
server_key_iv: "5db29d7fe5655f87ad6b85b59c60cef0"
server_key_version: "encryptionkey@0"
I20260924 10:43:55.510066 22880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:43:55.512372 22895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:55.513641 22880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:43:55.513762 22880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "e547aa30e8eb4de59ef9626bdfd2d400"
format_stamp: "Formatted at 2026-09-24 10:43:55 on dist-test-slave-j38h"
server_key: "aa53d880a60d203d598029963debbfb6"
server_key_iv: "5db29d7fe5655f87ad6b85b59c60cef0"
server_key_version: "encryptionkey@0"
I20260924 10:43:55.513863 22880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:55.518244 22880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:55.521546 22880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:55.521739 22880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:55.528703 22880 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44511
I20260924 10:43:55.528717 22947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44511 every 8 connection(s)
I20260924 10:43:55.529651 22880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:43:55.532420 22948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:55.537889 22948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400: Bootstrap starting.
I20260924 10:43:55.538439 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 22880
I20260924 10:43:55.538602 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:43:55.538872 2080 external_mini_cluster.cc:1428] Setting key 8079f2aa8c270a1773aa03bc17c1959c
I20260924 10:43:55.540653 22948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:55.541472 22948 log.cc:824] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:55.543536 22948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400: No bootstrap required, opened a new log
I20260924 10:43:55.544116 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:55.540136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38515 (local address 127.0.0.1:44511)
0924 10:43:55.540561 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:55.540575 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:55.540609 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:43:55.541322 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:55.541340 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:55.541686 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:55.542025 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:55.542839 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:55.542844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:55.542856 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:55.542866 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:55.542922 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:55.542943 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:55.543057 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:55.543212 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 10:43:55.543374 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":139,"threads_started":1}
I20260924 10:43:55.547086 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:43:55.547721 22948 raft_consensus.cc:348] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e547aa30e8eb4de59ef9626bdfd2d400" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44511 } }
I20260924 10:43:55.547968 22948 raft_consensus.cc:374] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:55.548046 22948 raft_consensus.cc:729] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e547aa30e8eb4de59ef9626bdfd2d400, State: Initialized, Role: FOLLOWER
I20260924 10:43:55.548555 22948 consensus_queue.cc:260] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e547aa30e8eb4de59ef9626bdfd2d400" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44511 } }
I20260924 10:43:55.548723 22948 raft_consensus.cc:388] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:43:55.548797 22948 raft_consensus.cc:482] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:43:55.548892 22948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:55.550172 22948 raft_consensus.cc:504] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e547aa30e8eb4de59ef9626bdfd2d400" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44511 } }
I20260924 10:43:55.550570 22948 leader_election.cc:302] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e547aa30e8eb4de59ef9626bdfd2d400; no voters:
I20260924 10:43:55.550946 22948 leader_election.cc:288] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:43:55.551077 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:55.549266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55685 (local address 127.0.0.1:44511)
0924 10:43:55.549374 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:55.549380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:55.549612 (+ 232us) server_negotiation.cc:407] Connection header received
0924 10:43:55.549729 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:55.549731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:55.549760 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:55.549824 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:55.550552 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:55.550555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:55.550557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:55.550567 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:55.550592 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:55.550601 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:55.550669 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:55.550828 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 10:43:55.550861 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 10:43:55.551025 22958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:55.551430 22958 raft_consensus.cc:686] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [term 1 LEADER]: Becoming Leader. State: Replica: e547aa30e8eb4de59ef9626bdfd2d400, State: Running, Role: LEADER
I20260924 10:43:55.551767 22958 consensus_queue.cc:237] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e547aa30e8eb4de59ef9626bdfd2d400" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44511 } }
I20260924 10:43:55.552345 22948 sys_catalog.cc:564] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:43:55.553160 22959 sys_catalog.cc:455] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e547aa30e8eb4de59ef9626bdfd2d400. Latest consensus state: current_term: 1 leader_uuid: "e547aa30e8eb4de59ef9626bdfd2d400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e547aa30e8eb4de59ef9626bdfd2d400" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44511 } } }
I20260924 10:43:55.553328 22959 sys_catalog.cc:458] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:55.553633 22960 sys_catalog.cc:455] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e547aa30e8eb4de59ef9626bdfd2d400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e547aa30e8eb4de59ef9626bdfd2d400" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44511 } } }
I20260924 10:43:55.553743 22960 sys_catalog.cc:458] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400 [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:55.553841 22967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:43:55.556937 22967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:43:55.562397 22967 catalog_manager.cc:1347] Generated new cluster ID: 29d8b0a44cff41cdb934aac60fedbe6c
I20260924 10:43:55.562484 22967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:43:55.587394 22967 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:43:55.587729 22967 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:43:55.595638 22967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e547aa30e8eb4de59ef9626bdfd2d400: Generated new TSK 0
I20260924 10:43:55.596503 22967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:43:55.601064 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 22880
2026-09-24T10:43:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:43:55 dist-test-slave-j38h krb5kdc[22979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:43:55 dist-test-slave-j38h krb5kdc[22979](info): set up 2 sockets
Sep 24 10:43:55 dist-test-slave-j38h krb5kdc[22979](info): commencing operation
krb5kdc: starting...
W20260924 10:43:56.665789 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:43:56 dist-test-slave-j38h krb5kdc[22979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:43:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:43:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:43:56.792207 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:56.903566 22992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:56.903867 22992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:56.903944 22992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:56.908046 22992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:43:56.908123 22992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:56.908159 22992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:56.908216 22992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:43:56.908254 22992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:43:56.913781 22992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:56.914850 22992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:56.915882 22992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:56.924518 23000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:56.924525 22998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:56.924566 22997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:56.925025 22992 server_base.cc:1034] running on GCE node
I20260924 10:43:56.925566 22992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:56.926700 22992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:56.927901 22992 hybrid_clock.cc:648] HybridClock initialized: now 1790246636927887 us; error 29 us; skew 500 ppm
Sep 24 10:43:56 dist-test-slave-j38h krb5kdc[22979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:56.944962 22992 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:43:56.946074 22992 webserver.cc:460] Webserver started at http://127.0.0.1:45971/ using document root <none> and password file <none>
I20260924 10:43:56.946678 22992 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:56.946770 22992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:56.947024 22992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:56.949247 22992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "85bb9b1aa68a4972bfb3a4ccd61a0403"
format_stamp: "Formatted at 2026-09-24 10:43:56 on dist-test-slave-j38h"
server_key: "816468d4905b5f5d4d444cfbb36557e7"
server_key_iv: "340afd5746a1aa19af1a79f894527fb5"
server_key_version: "encryptionkey@0"
I20260924 10:43:56.949815 22992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "85bb9b1aa68a4972bfb3a4ccd61a0403"
format_stamp: "Formatted at 2026-09-24 10:43:56 on dist-test-slave-j38h"
server_key: "816468d4905b5f5d4d444cfbb36557e7"
server_key_iv: "340afd5746a1aa19af1a79f894527fb5"
server_key_version: "encryptionkey@0"
I20260924 10:43:56.953593 22992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260924 10:43:56.955998 23007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:56.957186 22992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:43:56.957324 22992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "85bb9b1aa68a4972bfb3a4ccd61a0403"
format_stamp: "Formatted at 2026-09-24 10:43:56 on dist-test-slave-j38h"
server_key: "816468d4905b5f5d4d444cfbb36557e7"
server_key_iv: "340afd5746a1aa19af1a79f894527fb5"
server_key_version: "encryptionkey@0"
I20260924 10:43:56.957435 22992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:56.974352 22992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:56.977358 22992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:56.977560 22992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:56.984666 22992 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43711
I20260924 10:43:56.984702 23059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43711 every 8 connection(s)
I20260924 10:43:56.985657 22992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:43:56.988329 23060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:56.988848 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 22992
I20260924 10:43:56.989020 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:43:56.989317 2080 external_mini_cluster.cc:1428] Setting key ab4e42feba717577676e66d1994f7dcd
I20260924 10:43:56.995489 23060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403: Bootstrap starting.
I20260924 10:43:56.997998 23060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403: Neither blocks nor log segments found. Creating new log.
Sep 24 10:43:56 dist-test-slave-j38h krb5kdc[22979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:43:56.998911 23060 log.cc:824] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:57.001103 23060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403: No bootstrap required, opened a new log
I20260924 10:43:57.004009 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:56.990709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:47755 (local address 127.0.0.1:43711)
0924 10:43:56.991112 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:56.991133 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:56.991176 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:43:56.991908 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:56.991931 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:56.992368 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:56.992724 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:56.993446 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:56.994940 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:56.995812 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:56.996467 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:56.996625 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:56.999369 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:56.999375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:56.999391 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:56.999418 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:57.001193 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:57.001750 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:57.001754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:57.001758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:57.001859 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:57.002199 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:57.002201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:57.002202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:57.002582 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:57.002741 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:57.003065 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 10:43:57.003273 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260924 10:43:57.005101 23060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bb9b1aa68a4972bfb3a4ccd61a0403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } }
I20260924 10:43:57.005321 23060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:57.005393 23060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85bb9b1aa68a4972bfb3a4ccd61a0403, State: Initialized, Role: FOLLOWER
I20260924 10:43:57.005816 23060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bb9b1aa68a4972bfb3a4ccd61a0403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } }
I20260924 10:43:57.005980 23060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:43:57.006052 23060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:43:57.006147 23060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:57.007130 23060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bb9b1aa68a4972bfb3a4ccd61a0403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } }
I20260924 10:43:57.007158 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:43:57.007469 23060 leader_election.cc:302] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85bb9b1aa68a4972bfb3a4ccd61a0403; no voters:
I20260924 10:43:57.007786 23060 leader_election.cc:288] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:43:57.007866 23065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:57.008039 23065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [term 1 LEADER]: Becoming Leader. State: Replica: 85bb9b1aa68a4972bfb3a4ccd61a0403, State: Running, Role: LEADER
I20260924 10:43:57.008404 23065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bb9b1aa68a4972bfb3a4ccd61a0403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } }
I20260924 10:43:57.009097 23060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:43:57.010074 23066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85bb9b1aa68a4972bfb3a4ccd61a0403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bb9b1aa68a4972bfb3a4ccd61a0403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } } }
I20260924 10:43:57.010198 23066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:57.010598 23079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:43:57.010622 23067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85bb9b1aa68a4972bfb3a4ccd61a0403. Latest consensus state: current_term: 1 leader_uuid: "85bb9b1aa68a4972bfb3a4ccd61a0403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bb9b1aa68a4972bfb3a4ccd61a0403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43711 } } }
I20260924 10:43:57.011087 23067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403 [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:57.014097 23079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:43:57.019232 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:57.010587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:43375 (local address 127.0.0.1:43711)
0924 10:43:57.010692 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:57.010696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:57.010707 (+ 11us) server_negotiation.cc:407] Connection header received
0924 10:43:57.010734 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:57.010736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:57.010800 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:57.010879 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:57.011651 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:57.012402 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:57.013584 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:57.013805 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:57.013917 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:57.015322 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:57.015325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:57.015328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:43:57.015354 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:57.017272 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:57.017855 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:57.017859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:57.017860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:57.017952 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:43:57.018314 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:43:57.018317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:43:57.018319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:43:57.018523 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:57.018689 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:57.018975 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 10:43:57.019076 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 10:43:57.020848 23079 catalog_manager.cc:1347] Generated new cluster ID: 4938dd7a9fe742358051833c653fb2e9
I20260924 10:43:57.020919 23079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:43:57.053864 23079 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:43:57.055128 23079 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:43:57.074232 23079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85bb9b1aa68a4972bfb3a4ccd61a0403: Generated new TSK 0
I20260924 10:43:57.075039 23079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:43:57.087754 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 22992
W20260924 10:43:57.092684 22977 connection.cc:535] client connection to 127.0.0.1:43711 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43711): Connection reset by peer (error 104)
2026-09-24T10:43:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:43:57 dist-test-slave-j38h krb5kdc[23091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:43:57 dist-test-slave-j38h krb5kdc[23091](info): set up 2 sockets
Sep 24 10:43:57 dist-test-slave-j38h krb5kdc[23091](info): commencing operation
krb5kdc: starting...
W20260924 10:43:58.129236 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:43:58 dist-test-slave-j38h krb5kdc[23091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:43:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:43:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:43:58.249967 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34001
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:58.358330 23104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:58.358664 23104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:58.358741 23104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:58.362911 23104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:43:58.363020 23104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:58.363060 23104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:58.363123 23104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:43:58.363163 23104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:43:58.368613 23104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:58.369797 23104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:58.370851 23104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:58.379592 23109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:58.379632 23110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:58.379745 23104 server_base.cc:1034] running on GCE node
W20260924 10:43:58.379899 23112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:58.380540 23104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:58.381727 23104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:58.382917 23104 hybrid_clock.cc:648] HybridClock initialized: now 1790246638382903 us; error 26 us; skew 500 ppm
Sep 24 10:43:58 dist-test-slave-j38h krb5kdc[23091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:58.385718 23104 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:43:58.386600 23104 webserver.cc:460] Webserver started at http://127.0.0.1:35899/ using document root <none> and password file <none>
I20260924 10:43:58.387192 23104 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:58.387351 23104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:58.387591 23104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:58.389801 23104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "2539a26945b54586b23d32ed687a2a5a"
format_stamp: "Formatted at 2026-09-24 10:43:58 on dist-test-slave-j38h"
server_key: "559fd147343fc41df374248799b7c2cd"
server_key_iv: "3e275f619b00e59cfd77784975f8469c"
server_key_version: "encryptionkey@0"
I20260924 10:43:58.390420 23104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "2539a26945b54586b23d32ed687a2a5a"
format_stamp: "Formatted at 2026-09-24 10:43:58 on dist-test-slave-j38h"
server_key: "559fd147343fc41df374248799b7c2cd"
server_key_iv: "3e275f619b00e59cfd77784975f8469c"
server_key_version: "encryptionkey@0"
I20260924 10:43:58.394218 23104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:43:58.396807 23119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:58.398108 23104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:43:58.398257 23104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "2539a26945b54586b23d32ed687a2a5a"
format_stamp: "Formatted at 2026-09-24 10:43:58 on dist-test-slave-j38h"
server_key: "559fd147343fc41df374248799b7c2cd"
server_key_iv: "3e275f619b00e59cfd77784975f8469c"
server_key_version: "encryptionkey@0"
I20260924 10:43:58.398358 23104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:58.422331 23104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:58.425733 23104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:58.425946 23104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:58.433296 23104 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34001
I20260924 10:43:58.433312 23171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34001 every 8 connection(s)
I20260924 10:43:58.434316 23104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:43:58.436002 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 23104
I20260924 10:43:58.436161 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:43:58.436442 2080 external_mini_cluster.cc:1428] Setting key 7fb5fb6d1e15ee37d95e0eadb39de8e7
I20260924 10:43:58.437234 23172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:58.442902 23172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a: Bootstrap starting.
I20260924 10:43:58.445263 23172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:58.445492 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:58.437857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:56009 (local address 127.0.0.1:34001)
0924 10:43:58.438298 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:58.438314 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:58.438350 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:43:58.439037 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:58.439054 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:58.439382 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:58.439711 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:58.440125 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:58.441175 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:58.442062 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:58.442532 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:58.442696 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:58.443746 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:58.443751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:58.443764 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:58.444143 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:58.444194 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:58.444216 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:58.444359 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:58.444553 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 10:43:58.444690 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":138,"threads_started":1}
I20260924 10:43:58.446058 23172 log.cc:824] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:58.448045 23172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a: No bootstrap required, opened a new log
I20260924 10:43:58.448454 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:43:58.452193 23172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2539a26945b54586b23d32ed687a2a5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34001 } }
I20260924 10:43:58.452482 23172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:58.452549 23172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2539a26945b54586b23d32ed687a2a5a, State: Initialized, Role: FOLLOWER
I20260924 10:43:58.453035 23172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2539a26945b54586b23d32ed687a2a5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34001 } }
I20260924 10:43:58.453192 23172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:43:58.453284 23172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:43:58.453402 23172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:58.454638 23172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2539a26945b54586b23d32ed687a2a5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34001 } }
I20260924 10:43:58.455037 23172 leader_election.cc:302] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2539a26945b54586b23d32ed687a2a5a; no voters:
I20260924 10:43:58.455276 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:58.450695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:36597 (local address 127.0.0.1:34001)
0924 10:43:58.450839 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:58.450844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:58.451041 (+ 197us) server_negotiation.cc:407] Connection header received
0924 10:43:58.451152 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:58.451154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:58.451193 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:58.451263 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:58.451573 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:58.452116 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:58.453280 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:43:58.453657 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:43:58.453746 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:43:58.454707 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:58.454711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:58.454713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:58.454758 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:58.454792 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:58.454810 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:58.454906 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:58.455057 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 10:43:58.455113 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 10:43:58.455426 23172 leader_election.cc:288] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:43:58.455828 23183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:58.456053 23183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [term 1 LEADER]: Becoming Leader. State: Replica: 2539a26945b54586b23d32ed687a2a5a, State: Running, Role: LEADER
I20260924 10:43:58.456437 23183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2539a26945b54586b23d32ed687a2a5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34001 } }
I20260924 10:43:58.456759 23172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:43:58.458127 23184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2539a26945b54586b23d32ed687a2a5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2539a26945b54586b23d32ed687a2a5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34001 } } }
I20260924 10:43:58.458253 23184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:58.458472 23182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2539a26945b54586b23d32ed687a2a5a. Latest consensus state: current_term: 1 leader_uuid: "2539a26945b54586b23d32ed687a2a5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2539a26945b54586b23d32ed687a2a5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34001 } } }
I20260924 10:43:58.458642 23182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:58.458634 23192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:43:58.461786 23192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:43:58.467869 23192 catalog_manager.cc:1347] Generated new cluster ID: 415d12b7afe042c0aa74007c32a1bb3f
I20260924 10:43:58.467970 23192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:43:58.476075 23192 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:43:58.477301 23192 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:43:58.493245 23192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2539a26945b54586b23d32ed687a2a5a: Generated new TSK 0
I20260924 10:43:58.494105 23192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:43:58.507480 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 23104
W20260924 10:43:58.513355 23089 connection.cc:535] client connection to 127.0.0.1:34001 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34001): Connection reset by peer (error 104)
2026-09-24T10:43:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1420 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:43:58 dist-test-slave-j38h krb5kdc[23203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:43:58 dist-test-slave-j38h krb5kdc[23203](info): set up 2 sockets
Sep 24 10:43:58 dist-test-slave-j38h krb5kdc[23203](info): commencing operation
krb5kdc: starting...
W20260924 10:43:59.553401 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:43:59 dist-test-slave-j38h krb5kdc[23203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:43:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:43:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:43:59.674067 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34851
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:43:59.782317 23216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:43:59.782635 23216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:43:59.782720 23216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:43:59.786656 23216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:43:59.786753 23216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:43:59.786790 23216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:43:59.786829 23216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:43:59.786861 23216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:43:59.792081 23216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:43:59.793213 23216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:43:59.794306 23216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:43:59.802943 23224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:59.802940 23221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:43:59.803150 23222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:43:59.803498 23216 server_base.cc:1034] running on GCE node
I20260924 10:43:59.804091 23216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:43:59.805410 23216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:43:59.806612 23216 hybrid_clock.cc:648] HybridClock initialized: now 1790246639806581 us; error 41 us; skew 500 ppm
Sep 24 10:43:59 dist-test-slave-j38h krb5kdc[23203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:43:59.809527 23216 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:43:59.810488 23216 webserver.cc:460] Webserver started at http://127.0.0.1:38517/ using document root <none> and password file <none>
I20260924 10:43:59.811065 23216 fs_manager.cc:359] Metadata directory not provided
I20260924 10:43:59.811127 23216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:43:59.811342 23216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:43:59.813477 23216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "66a35c247068478a8f13186163b0f682"
format_stamp: "Formatted at 2026-09-24 10:43:59 on dist-test-slave-j38h"
server_key: "bf3ac0ee40bb1d10d9a1ea22e1fd2ca2"
server_key_iv: "5d77bfbc3a2eb34fc4150343105affe0"
server_key_version: "encryptionkey@0"
I20260924 10:43:59.814039 23216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "66a35c247068478a8f13186163b0f682"
format_stamp: "Formatted at 2026-09-24 10:43:59 on dist-test-slave-j38h"
server_key: "bf3ac0ee40bb1d10d9a1ea22e1fd2ca2"
server_key_iv: "5d77bfbc3a2eb34fc4150343105affe0"
server_key_version: "encryptionkey@0"
I20260924 10:43:59.817813 23216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 10:43:59.820417 23231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:43:59.821800 23216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 10:43:59.822038 23216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "66a35c247068478a8f13186163b0f682"
format_stamp: "Formatted at 2026-09-24 10:43:59 on dist-test-slave-j38h"
server_key: "bf3ac0ee40bb1d10d9a1ea22e1fd2ca2"
server_key_iv: "5d77bfbc3a2eb34fc4150343105affe0"
server_key_version: "encryptionkey@0"
I20260924 10:43:59.822157 23216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:43:59.826845 23216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:43:59.829986 23216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:43:59.830173 23216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:43:59.837196 23216 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34851
I20260924 10:43:59.837225 23283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34851 every 8 connection(s)
I20260924 10:43:59.838142 23216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:43:59.840020 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 23216
I20260924 10:43:59.840158 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:43:59.840461 2080 external_mini_cluster.cc:1428] Setting key 9510eac46a91373af38bc008cbd70688
I20260924 10:43:59.841225 23284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:43:59.845722 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:59.841915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:32953 (local address 127.0.0.1:34851)
0924 10:43:59.842336 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:59.842351 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:59.842386 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:43:59.843015 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:59.843032 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:59.843303 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:59.843655 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:59.844476 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:59.844481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:59.844494 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:59.844504 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:59.844561 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:59.844581 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:59.844734 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:59.844809 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 10:43:59.844970 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":132,"threads_started":1}
I20260924 10:43:59.847424 23284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682: Bootstrap starting.
I20260924 10:43:59.848582 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:43:59.850206 23284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682: Neither blocks nor log segments found. Creating new log.
I20260924 10:43:59.851100 23284 log.cc:824] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682: Log is configured to *not* fsync() on all Append() calls
I20260924 10:43:59.852744 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:43:59.850951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:59545 (local address 127.0.0.1:34851)
0924 10:43:59.851074 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:43:59.851078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:43:59.851302 (+ 224us) server_negotiation.cc:407] Connection header received
0924 10:43:59.851517 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:43:59.851520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:43:59.851562 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:43:59.851645 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:43:59.852343 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:43:59.852346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:43:59.852349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:43:59.852361 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:43:59.852388 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:43:59.852407 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:43:59.852499 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 10:43:59.852584 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 10:43:59.852625 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:43:59.853247 23284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682: No bootstrap required, opened a new log
I20260924 10:43:59.856734 23284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a35c247068478a8f13186163b0f682" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } }
I20260924 10:43:59.856951 23284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:43:59.857054 23284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a35c247068478a8f13186163b0f682, State: Initialized, Role: FOLLOWER
I20260924 10:43:59.857535 23284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a35c247068478a8f13186163b0f682" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } }
I20260924 10:43:59.857686 23284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:43:59.857769 23284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:43:59.857860 23284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:43:59.858767 23284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a35c247068478a8f13186163b0f682" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } }
I20260924 10:43:59.859110 23284 leader_election.cc:302] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66a35c247068478a8f13186163b0f682; no voters:
I20260924 10:43:59.859411 23284 leader_election.cc:288] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:43:59.859535 23294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:43:59.859781 23294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [term 1 LEADER]: Becoming Leader. State: Replica: 66a35c247068478a8f13186163b0f682, State: Running, Role: LEADER
I20260924 10:43:59.860122 23294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a35c247068478a8f13186163b0f682" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } }
I20260924 10:43:59.860585 23284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:43:59.861719 23295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66a35c247068478a8f13186163b0f682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a35c247068478a8f13186163b0f682" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } } }
I20260924 10:43:59.862149 23295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [sys.catalog]: This master's current role is: LEADER
I20260924 10:43:59.862036 23296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66a35c247068478a8f13186163b0f682. Latest consensus state: current_term: 1 leader_uuid: "66a35c247068478a8f13186163b0f682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a35c247068478a8f13186163b0f682" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34851 } } }
I20260924 10:43:59.863010 23296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682 [sys.catalog]: This master's current role is: LEADER
W20260924 10:43:59.865396 23308 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:43:59.865458 23308 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:43:59.865532 23309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:43:59.866231 23309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:43:59.871949 23309 catalog_manager.cc:1347] Generated new cluster ID: 839d1c77c2bd4059b2b725a6167a5dab
I20260924 10:43:59.872025 23309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:43:59.887773 23309 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:43:59.888132 23309 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:43:59.903146 23309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66a35c247068478a8f13186163b0f682: Generated new TSK 0
I20260924 10:43:59.904011 23309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:43:59.918668 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 23216
2026-09-24T10:43:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:43:59 dist-test-slave-j38h krb5kdc[23315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:43:59 dist-test-slave-j38h krb5kdc[23315](info): set up 2 sockets
Sep 24 10:43:59 dist-test-slave-j38h krb5kdc[23315](info): commencing operation
krb5kdc: starting...
W20260924 10:44:00.957719 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:44:01 dist-test-slave-j38h krb5kdc[23315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:44:01.080433 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:01.190752 23328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:01.191025 23328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:01.191066 23328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:01.195058 23328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:01.195125 23328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:01.195144 23328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:01.195161 23328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:01.195212 23328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:01.200438 23328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45987
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:01.201503 23328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:01.202515 23328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:01.210920 23336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:01.210934 23333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:01.210934 23334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:01.211171 23328 server_base.cc:1034] running on GCE node
I20260924 10:44:01.211902 23328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:01.213119 23328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:01.214303 23328 hybrid_clock.cc:648] HybridClock initialized: now 1790246641214287 us; error 38 us; skew 500 ppm
Sep 24 10:44:01 dist-test-slave-j38h krb5kdc[23315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:01.216962 23328 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:44:01.217833 23328 webserver.cc:460] Webserver started at http://127.0.0.1:46071/ using document root <none> and password file <none>
I20260924 10:44:01.218551 23328 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:01.218632 23328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:01.218839 23328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:01.220866 23328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "1dffd8a6f3e94580a80aed424122f640"
format_stamp: "Formatted at 2026-09-24 10:44:01 on dist-test-slave-j38h"
server_key: "5c103e1f465fda631402b3d6aae198f4"
server_key_iv: "fb959e9278d2ecc54f0681ab13e005d4"
server_key_version: "encryptionkey@0"
I20260924 10:44:01.221503 23328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "1dffd8a6f3e94580a80aed424122f640"
format_stamp: "Formatted at 2026-09-24 10:44:01 on dist-test-slave-j38h"
server_key: "5c103e1f465fda631402b3d6aae198f4"
server_key_iv: "fb959e9278d2ecc54f0681ab13e005d4"
server_key_version: "encryptionkey@0"
I20260924 10:44:01.225121 23328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:44:01.227420 23343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:01.228601 23328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:01.228734 23328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "1dffd8a6f3e94580a80aed424122f640"
format_stamp: "Formatted at 2026-09-24 10:44:01 on dist-test-slave-j38h"
server_key: "5c103e1f465fda631402b3d6aae198f4"
server_key_iv: "fb959e9278d2ecc54f0681ab13e005d4"
server_key_version: "encryptionkey@0"
I20260924 10:44:01.228843 23328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:01.254580 23328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:01.257831 23328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:01.258056 23328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:01.265091 23328 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45987
I20260924 10:44:01.265130 23395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45987 every 8 connection(s)
I20260924 10:44:01.266050 23328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:01.266987 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 23328
I20260924 10:44:01.267138 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:44:01.267458 2080 external_mini_cluster.cc:1428] Setting key 763a14356c75f0493e2899fc80cbb2de
I20260924 10:44:01.269443 23396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:01.274996 23396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640: Bootstrap starting.
Sep 24 10:44:01 dist-test-slave-j38h krb5kdc[23315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 10:44:01.277498 23396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:01.278378 23396 log.cc:824] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:01.280275 23396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640: No bootstrap required, opened a new log
I20260924 10:44:01.282284 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:01.268903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:36717 (local address 127.0.0.1:45987)
0924 10:44:01.269366 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:01.269388 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:01.269422 (+ 34us) server_negotiation.cc:407] Connection header received
0924 10:44:01.270088 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:01.270105 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:01.270440 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:01.270785 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:01.271573 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:01.272541 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:01.273485 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:01.273956 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:01.274126 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:01.277211 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:01.277224 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:01.277238 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:01.277315 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:01.279239 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:01.279805 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:01.279811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:01.279816 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:01.279924 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:01.280345 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:01.280348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:01.280349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:01.280759 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:01.280980 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:01.281329 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 10:44:01.281548 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":157,"threads_started":1}
I20260924 10:44:01.284227 23396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dffd8a6f3e94580a80aed424122f640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45987 } }
I20260924 10:44:01.284480 23396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:01.284544 23396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dffd8a6f3e94580a80aed424122f640, State: Initialized, Role: FOLLOWER
I20260924 10:44:01.285094 23396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dffd8a6f3e94580a80aed424122f640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45987 } }
I20260924 10:44:01.285265 23396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:01.285318 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:44:01.285333 23396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:01.285436 23396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:01.286602 23396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dffd8a6f3e94580a80aed424122f640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45987 } }
I20260924 10:44:01.287002 23396 leader_election.cc:302] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dffd8a6f3e94580a80aed424122f640; no voters:
I20260924 10:44:01.287355 23396 leader_election.cc:288] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:01.287465 23405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:01.287793 23405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [term 1 LEADER]: Becoming Leader. State: Replica: 1dffd8a6f3e94580a80aed424122f640, State: Running, Role: LEADER
I20260924 10:44:01.288108 23405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dffd8a6f3e94580a80aed424122f640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45987 } }
I20260924 10:44:01.288617 23396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:01.289872 23406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dffd8a6f3e94580a80aed424122f640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dffd8a6f3e94580a80aed424122f640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45987 } } }
I20260924 10:44:01.290000 23406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:01.290319 23408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dffd8a6f3e94580a80aed424122f640. Latest consensus state: current_term: 1 leader_uuid: "1dffd8a6f3e94580a80aed424122f640" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dffd8a6f3e94580a80aed424122f640" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45987 } } }
I20260924 10:44:01.290446 23408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640 [sys.catalog]: This master's current role is: LEADER
W20260924 10:44:01.294250 23421 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:44:01.294373 23421 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:44:01.294464 23416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:01.295357 23416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:01.299401 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:01.287673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:48703 (local address 127.0.0.1:45987)
0924 10:44:01.287773 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:01.287777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:01.291000 (+ 3223us) server_negotiation.cc:407] Connection header received
0924 10:44:01.291220 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:01.291223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:01.291283 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:01.291339 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:01.292380 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:01.292958 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:01.293847 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:01.294094 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:01.294205 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:01.295718 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:01.295722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:01.295725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:01.295751 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:01.297297 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:01.297967 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:01.297970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:01.297972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:01.298058 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:01.298454 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:01.298457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:01.298459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:01.298664 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:01.298831 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:01.299162 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 10:44:01.299262 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:44:01.301707 23416 catalog_manager.cc:1347] Generated new cluster ID: 0604fb50eba2498fa16b2c3f6a03dce9
I20260924 10:44:01.301782 23416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:01.316581 23416 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:01.318069 23416 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:01.330549 23416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dffd8a6f3e94580a80aed424122f640: Generated new TSK 0
I20260924 10:44:01.331357 23416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:44:01.340035 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 23328
W20260924 10:44:01.345973 23313 connection.cc:535] client connection to 127.0.0.1:45987 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45987): Connection reset by peer (error 104)
2026-09-24T10:44:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:01 dist-test-slave-j38h krb5kdc[23427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:01 dist-test-slave-j38h krb5kdc[23427](info): set up 2 sockets
Sep 24 10:44:01 dist-test-slave-j38h krb5kdc[23427](info): commencing operation
krb5kdc: starting...
W20260924 10:44:02.421311 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:44:02 dist-test-slave-j38h krb5kdc[23427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:44:02.548406 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:02.658885 23440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:02.659219 23440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:02.659301 23440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:02.663897 23440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:02.664033 23440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:02.664077 23440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:02.664135 23440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:02.664173 23440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:02.669770 23440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37535
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:02.670990 23440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:02.672093 23440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:44:02.680907 23440 server_base.cc:1034] running on GCE node
W20260924 10:44:02.680804 23446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:02.680809 23445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:02.680828 23448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:02.681839 23440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:02.683018 23440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:02.684201 23440 hybrid_clock.cc:648] HybridClock initialized: now 1790246642684177 us; error 35 us; skew 500 ppm
Sep 24 10:44:02 dist-test-slave-j38h krb5kdc[23427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:02.696918 23440 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:44:02.698060 23440 webserver.cc:460] Webserver started at http://127.0.0.1:44149/ using document root <none> and password file <none>
I20260924 10:44:02.698684 23440 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:02.698750 23440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:02.699002 23440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:02.701349 23440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "1fbe0bd3dcab437291ef3fb31b8ab777"
format_stamp: "Formatted at 2026-09-24 10:44:02 on dist-test-slave-j38h"
server_key: "adb16f58e45d79976d5e02b5905b7be6"
server_key_iv: "6bc91aae1bd8527f78e333f0167c9ff2"
server_key_version: "encryptionkey@0"
I20260924 10:44:02.701949 23440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "1fbe0bd3dcab437291ef3fb31b8ab777"
format_stamp: "Formatted at 2026-09-24 10:44:02 on dist-test-slave-j38h"
server_key: "adb16f58e45d79976d5e02b5905b7be6"
server_key_iv: "6bc91aae1bd8527f78e333f0167c9ff2"
server_key_version: "encryptionkey@0"
I20260924 10:44:02.706040 23440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 10:44:02.708865 23455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:02.710342 23440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 10:44:02.710487 23440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "1fbe0bd3dcab437291ef3fb31b8ab777"
format_stamp: "Formatted at 2026-09-24 10:44:02 on dist-test-slave-j38h"
server_key: "adb16f58e45d79976d5e02b5905b7be6"
server_key_iv: "6bc91aae1bd8527f78e333f0167c9ff2"
server_key_version: "encryptionkey@0"
I20260924 10:44:02.710606 23440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:02.722922 23440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:02.726205 23440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:02.726457 23440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:02.734558 23507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37535 every 8 connection(s)
I20260924 10:44:02.734555 23440 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37535
I20260924 10:44:02.735699 23440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:02.738584 23508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:02.744236 23508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777: Bootstrap starting.
I20260924 10:44:02.745319 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 23440
I20260924 10:44:02.745492 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:44:02.745795 2080 external_mini_cluster.cc:1428] Setting key 879b4572ce7753bd4774289fba7151cc
I20260924 10:44:02.747460 23508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:02.748256 23508 log.cc:824] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:02.750597 23508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777: No bootstrap required, opened a new log
I20260924 10:44:02.755451 23508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbe0bd3dcab437291ef3fb31b8ab777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } }
I20260924 10:44:02.755739 23508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:02.755834 23508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fbe0bd3dcab437291ef3fb31b8ab777, State: Initialized, Role: FOLLOWER
I20260924 10:44:02.756142 23511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:02.747010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:51135 (local address 127.0.0.1:37535)
0924 10:44:02.747464 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:02.747498 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:02.747539 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:44:02.748298 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:02.748325 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:02.748831 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:02.749274 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:02.749731 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:02.750929 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:02.752241 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:02.752806 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:02.752943 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:02.753999 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:02.754005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:02.754020 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:02.754461 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:02.754520 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:02.754545 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:02.754656 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:02.754925 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 10:44:02.755094 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":146,"threads_started":1}
I20260924 10:44:02.756412 23508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbe0bd3dcab437291ef3fb31b8ab777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } }
I20260924 10:44:02.756589 23508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:02.756664 23508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:02.756769 23508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:02.757947 23508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbe0bd3dcab437291ef3fb31b8ab777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } }
I20260924 10:44:02.758338 23508 leader_election.cc:302] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fbe0bd3dcab437291ef3fb31b8ab777; no voters:
I20260924 10:44:02.758745 23508 leader_election.cc:288] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:02.759043 23513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:02.759407 23513 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [term 1 LEADER]: Becoming Leader. State: Replica: 1fbe0bd3dcab437291ef3fb31b8ab777, State: Running, Role: LEADER
I20260924 10:44:02.759656 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:44:02.759814 23513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbe0bd3dcab437291ef3fb31b8ab777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } }
I20260924 10:44:02.760201 23508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:02.761293 23513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fbe0bd3dcab437291ef3fb31b8ab777. Latest consensus state: current_term: 1 leader_uuid: "1fbe0bd3dcab437291ef3fb31b8ab777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbe0bd3dcab437291ef3fb31b8ab777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } } }
I20260924 10:44:02.761514 23513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:02.761770 23514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fbe0bd3dcab437291ef3fb31b8ab777" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbe0bd3dcab437291ef3fb31b8ab777" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37535 } } }
I20260924 10:44:02.761876 23514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:02.761993 23526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:02.765648 23526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:02.768530 23511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:02.762302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:60103 (local address 127.0.0.1:37535)
0924 10:44:02.762418 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:02.762423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:02.763079 (+ 656us) server_negotiation.cc:407] Connection header received
0924 10:44:02.763330 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:02.763334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:02.763399 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:02.763485 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:02.763979 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:02.764840 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:02.766142 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:02.766530 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:02.766637 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:02.767863 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:02.767867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:02.767876 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:02.767892 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:02.767920 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:02.767935 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:02.768019 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:02.768389 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 10:44:02.768421 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:44:02.772795 23526 catalog_manager.cc:1347] Generated new cluster ID: b80b8c1c49d54abfa0860c5b3054d444
I20260924 10:44:02.772868 23526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:02.784735 23526 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:02.786132 23526 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:02.794103 23526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fbe0bd3dcab437291ef3fb31b8ab777: Generated new TSK 0
I20260924 10:44:02.795050 23526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:44:02.802359 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 23440
W20260924 10:44:02.807704 23425 connection.cc:535] client connection to 127.0.0.1:37535 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37535): Connection reset by peer (error 104)
2026-09-24T10:44:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1461 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:02 dist-test-slave-j38h krb5kdc[23539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:02 dist-test-slave-j38h krb5kdc[23539](info): set up 2 sockets
Sep 24 10:44:02 dist-test-slave-j38h krb5kdc[23539](info): commencing operation
krb5kdc: starting...
W20260924 10:44:03.841784 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:44:03 dist-test-slave-j38h krb5kdc[23539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 10:44:03.966245 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44965
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:04.075119 23552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:04.075462 23552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:04.075555 23552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:04.079701 23552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:04.079797 23552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:04.079833 23552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:04.079892 23552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:04.079928 23552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:04.085496 23552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44965
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:04.086647 23552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:04.087664 23552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:04.096213 23557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:04.096269 23560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:04.096393 23552 server_base.cc:1034] running on GCE node
W20260924 10:44:04.096189 23558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:04.097119 23552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:04.098201 23552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:04.099376 23552 hybrid_clock.cc:648] HybridClock initialized: now 1790246644099353 us; error 36 us; skew 500 ppm
Sep 24 10:44:04 dist-test-slave-j38h krb5kdc[23539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:04.102108 23552 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 10:44:04.102962 23552 webserver.cc:460] Webserver started at http://127.0.0.1:33695/ using document root <none> and password file <none>
I20260924 10:44:04.103506 23552 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:04.103586 23552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:04.103816 23552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:04.105870 23552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "6aba263dc0c54e4f83bdceff59aadf42"
format_stamp: "Formatted at 2026-09-24 10:44:04 on dist-test-slave-j38h"
server_key: "864b2158b55b1747cdc84d455c2f0960"
server_key_iv: "c396082f49cc135b880f9169d5533e77"
server_key_version: "encryptionkey@0"
I20260924 10:44:04.106439 23552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "6aba263dc0c54e4f83bdceff59aadf42"
format_stamp: "Formatted at 2026-09-24 10:44:04 on dist-test-slave-j38h"
server_key: "864b2158b55b1747cdc84d455c2f0960"
server_key_iv: "c396082f49cc135b880f9169d5533e77"
server_key_version: "encryptionkey@0"
I20260924 10:44:04.110044 23552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:44:04.112375 23567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:04.113619 23552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:04.113763 23552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "6aba263dc0c54e4f83bdceff59aadf42"
format_stamp: "Formatted at 2026-09-24 10:44:04 on dist-test-slave-j38h"
server_key: "864b2158b55b1747cdc84d455c2f0960"
server_key_iv: "c396082f49cc135b880f9169d5533e77"
server_key_version: "encryptionkey@0"
I20260924 10:44:04.113876 23552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:04.118337 23552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:04.121196 23552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:04.121376 23552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:04.128408 23552 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44965
I20260924 10:44:04.128440 23619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44965 every 8 connection(s)
I20260924 10:44:04.129350 23552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:04.132040 23620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:04.132413 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 23552
I20260924 10:44:04.132563 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:44:04.132858 2080 external_mini_cluster.cc:1428] Setting key ac610b729f713d6de7e2676f7605234a
I20260924 10:44:04.138486 23620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42: Bootstrap starting.
I20260924 10:44:04.138703 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:04.134330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:57679 (local address 127.0.0.1:44965)
0924 10:44:04.134838 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:04.134855 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:04.134896 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:44:04.135621 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:04.135640 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:04.135957 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:04.136357 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:04.137296 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:04.137301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:04.137315 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:04.137329 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:04.137394 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:04.137417 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:04.137598 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:04.137680 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 10:44:04.137878 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":126,"threads_started":1}
I20260924 10:44:04.141172 23620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:04.141845 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:44:04.142067 23620 log.cc:824] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:04.144313 23620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42: No bootstrap required, opened a new log
I20260924 10:44:04.145848 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:04.143957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.70:48079 (local address 127.0.0.1:44965)
0924 10:44:04.144071 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:04.144074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:04.144340 (+ 266us) server_negotiation.cc:407] Connection header received
0924 10:44:04.144497 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:04.144500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:04.144530 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:04.144598 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:04.145421 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:04.145425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:04.145427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:04.145437 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:04.145461 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:04.145471 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:04.145540 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:04.145659 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 10:44:04.145693 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:44:04.148761 23620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba263dc0c54e4f83bdceff59aadf42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44965 } }
I20260924 10:44:04.149036 23620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:04.149106 23620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aba263dc0c54e4f83bdceff59aadf42, State: Initialized, Role: FOLLOWER
I20260924 10:44:04.149569 23620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba263dc0c54e4f83bdceff59aadf42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44965 } }
I20260924 10:44:04.149716 23620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:04.149782 23620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:04.149874 23620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:04.150817 23620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba263dc0c54e4f83bdceff59aadf42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44965 } }
I20260924 10:44:04.151171 23620 leader_election.cc:302] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aba263dc0c54e4f83bdceff59aadf42; no voters:
I20260924 10:44:04.151521 23620 leader_election.cc:288] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:04.151597 23630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:04.151801 23630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [term 1 LEADER]: Becoming Leader. State: Replica: 6aba263dc0c54e4f83bdceff59aadf42, State: Running, Role: LEADER
I20260924 10:44:04.152146 23630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba263dc0c54e4f83bdceff59aadf42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44965 } }
I20260924 10:44:04.152875 23620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:04.154001 23632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aba263dc0c54e4f83bdceff59aadf42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba263dc0c54e4f83bdceff59aadf42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44965 } } }
I20260924 10:44:04.154040 23631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aba263dc0c54e4f83bdceff59aadf42. Latest consensus state: current_term: 1 leader_uuid: "6aba263dc0c54e4f83bdceff59aadf42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba263dc0c54e4f83bdceff59aadf42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44965 } } }
I20260924 10:44:04.154196 23631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:04.154192 23632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:04.154657 23639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:04.157975 23639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:04.163944 23639 catalog_manager.cc:1347] Generated new cluster ID: 30f681e983ff4650bf531490c449a283
I20260924 10:44:04.164023 23639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:04.176255 23639 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:04.176586 23639 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:04.181347 23639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6aba263dc0c54e4f83bdceff59aadf42: Generated new TSK 0
I20260924 10:44:04.182021 23639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:44:04.192710 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 23552
2026-09-24T10:44:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:04 dist-test-slave-j38h krb5kdc[23651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:04 dist-test-slave-j38h krb5kdc[23651](info): set up 2 sockets
Sep 24 10:44:04 dist-test-slave-j38h krb5kdc[23651](info): commencing operation
krb5kdc: starting...
W20260924 10:44:05.249300 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:44:05 dist-test-slave-j38h krb5kdc[23651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:44:05.376644 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:44615
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:38401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:44615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:05.486200 23664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:05.486474 23664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:05.486516 23664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:05.490475 23664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:05.490537 23664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:05.490556 23664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:05.490573 23664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:05.490588 23664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:05.495820 23664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:38401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:44615
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:44615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:05.496824 23664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:05.497874 23664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:05.506147 23672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:05.506165 23670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:05.506147 23669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:05.506469 23664 server_base.cc:1034] running on GCE node
I20260924 10:44:05.507061 23664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:05.508152 23664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:05.509379 23664 hybrid_clock.cc:648] HybridClock initialized: now 1790246645509353 us; error 39 us; skew 500 ppm
Sep 24 10:44:05 dist-test-slave-j38h krb5kdc[23651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246645, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:05.512244 23664 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:44:05.513198 23664 webserver.cc:460] Webserver started at http://127.2.8.62:40757/ using document root <none> and password file <none>
I20260924 10:44:05.513767 23664 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:05.513856 23664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:05.514083 23664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:05.516091 23664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "c342a67250774f778a743c1919f4c44c"
format_stamp: "Formatted at 2026-09-24 10:44:05 on dist-test-slave-j38h"
server_key: "19ecbabaf5fe6d5993db2bb3e323242e"
server_key_iv: "145b201066152cd1460637891808eb98"
server_key_version: "encryptionkey@0"
I20260924 10:44:05.516655 23664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "c342a67250774f778a743c1919f4c44c"
format_stamp: "Formatted at 2026-09-24 10:44:05 on dist-test-slave-j38h"
server_key: "19ecbabaf5fe6d5993db2bb3e323242e"
server_key_iv: "145b201066152cd1460637891808eb98"
server_key_version: "encryptionkey@0"
I20260924 10:44:05.520298 23664 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:44:05.522809 23679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:05.523993 23664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 10:44:05.524125 23664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "c342a67250774f778a743c1919f4c44c"
format_stamp: "Formatted at 2026-09-24 10:44:05 on dist-test-slave-j38h"
server_key: "19ecbabaf5fe6d5993db2bb3e323242e"
server_key_iv: "145b201066152cd1460637891808eb98"
server_key_version: "encryptionkey@0"
I20260924 10:44:05.524231 23664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:05.546398 23664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:05.549453 23664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:05.549660 23664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:05.556941 23664 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:44615
I20260924 10:44:05.556967 23731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:44615 every 8 connection(s)
I20260924 10:44:05.558396 23664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:05.561357 23732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:05.563027 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 23664
I20260924 10:44:05.563207 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:44:05.563509 2080 external_mini_cluster.cc:1428] Setting key 33c69090dfd44773b9f10199c9090e04
I20260924 10:44:05.568286 23732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c: Bootstrap starting.
I20260924 10:44:05.570848 23732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:05.571578 23732 log.cc:824] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c: Log is configured to *not* fsync() on all Append() calls
Sep 24 10:44:05 dist-test-slave-j38h krb5kdc[23651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:44:05.573838 23732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c: No bootstrap required, opened a new log
I20260924 10:44:05.577690 23732 raft_consensus.cc:348] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c342a67250774f778a743c1919f4c44c" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44615 } }
I20260924 10:44:05.577919 23732 raft_consensus.cc:374] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:05.577991 23732 raft_consensus.cc:729] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c342a67250774f778a743c1919f4c44c, State: Initialized, Role: FOLLOWER
I20260924 10:44:05.578467 23732 consensus_queue.cc:260] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c342a67250774f778a743c1919f4c44c" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44615 } }
I20260924 10:44:05.578586 23732 raft_consensus.cc:388] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:05.578670 23732 raft_consensus.cc:482] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:05.578752 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:05.564908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52247 (local address 127.2.8.62:44615)
0924 10:44:05.565373 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:05.565393 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:05.565434 (+ 41us) server_negotiation.cc:407] Connection header received
0924 10:44:05.566258 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:05.566284 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:05.566706 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:05.567087 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:05.567727 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:05.569196 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:05.570045 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:05.570704 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:05.570869 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:05.573848 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:05.573854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:05.573866 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:05.573918 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:05.575646 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:05.576208 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:05.576213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:05.576217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:05.576319 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:05.576641 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:05.576643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:05.576645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:05.577090 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:05.577259 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:05.577567 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 10:44:05.577812 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":114,"threads_started":1}
I20260924 10:44:05.578784 23732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:05.579689 23732 raft_consensus.cc:504] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c342a67250774f778a743c1919f4c44c" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44615 } }
I20260924 10:44:05.580052 23732 leader_election.cc:302] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c342a67250774f778a743c1919f4c44c; no voters:
I20260924 10:44:05.580353 23732 leader_election.cc:288] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:05.580461 23737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:05.580674 23737 raft_consensus.cc:686] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [term 1 LEADER]: Becoming Leader. State: Replica: c342a67250774f778a743c1919f4c44c, State: Running, Role: LEADER
I20260924 10:44:05.581027 23737 consensus_queue.cc:237] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c342a67250774f778a743c1919f4c44c" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44615 } }
I20260924 10:44:05.581663 23732 sys_catalog.cc:564] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:05.581988 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:44:05.582206 23739 sys_catalog.cc:455] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c342a67250774f778a743c1919f4c44c. Latest consensus state: current_term: 1 leader_uuid: "c342a67250774f778a743c1919f4c44c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c342a67250774f778a743c1919f4c44c" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44615 } } }
I20260924 10:44:05.582319 23739 sys_catalog.cc:458] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:05.582633 23738 sys_catalog.cc:455] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c342a67250774f778a743c1919f4c44c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c342a67250774f778a743c1919f4c44c" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 44615 } } }
I20260924 10:44:05.582718 23738 sys_catalog.cc:458] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c [sys.catalog]: This master's current role is: LEADER
W20260924 10:44:05.587203 23756 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:44:05.587287 23756 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:44:05.588076 23743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:05.588971 23743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:05.594223 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:05.584563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50937 (local address 127.2.8.62:44615)
0924 10:44:05.584671 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:05.584677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:05.585334 (+ 657us) server_negotiation.cc:407] Connection header received
0924 10:44:05.585407 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:05.585410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:05.585468 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:05.585555 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:05.586406 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:05.587148 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:05.588054 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:05.588383 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:05.588490 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:05.590194 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:05.590199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:05.590205 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:05.590240 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:05.591944 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:05.592485 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:05.592489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:05.592491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:05.592574 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:05.592919 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:05.592922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:05.592924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:05.593171 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:05.593340 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:05.593968 (+ 628us) server_negotiation.cc:299] Negotiation successful
0924 10:44:05.594068 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 10:44:05.595242 23743 catalog_manager.cc:1347] Generated new cluster ID: a33ff6ffe6c741f191c959ec34fd2cfa
I20260924 10:44:05.595296 23743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:05.613443 23743 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:05.614725 23743 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:05.638208 23743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c342a67250774f778a743c1919f4c44c: Generated new TSK 0
I20260924 10:44:05.639158 23743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:44:05.647465 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 23664
W20260924 10:44:05.652942 23649 connection.cc:535] client connection to 127.2.8.62:44615 recv error: Network error: failed to read from TLS socket (remote: 127.2.8.62:44615): Connection reset by peer (error 104)
2026-09-24T10:44:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:05 dist-test-slave-j38h krb5kdc[23763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:05 dist-test-slave-j38h krb5kdc[23763](info): set up 2 sockets
Sep 24 10:44:05 dist-test-slave-j38h krb5kdc[23763](info): commencing operation
krb5kdc: starting...
W20260924 10:44:06.685379 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:44:06 dist-test-slave-j38h krb5kdc[23763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:44:06.812077 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:42889
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:46617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:42889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:06.925820 23776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:06.926156 23776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:06.926239 23776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:06.930418 23776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:06.930516 23776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:06.930554 23776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:06.930614 23776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:06.930652 23776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:06.935971 23776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:46617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:42889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:42889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:06.937091 23776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:06.938305 23776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:06.946929 23784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:06.946971 23782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:06.946991 23776 server_base.cc:1034] running on GCE node
W20260924 10:44:06.947139 23781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:06.947845 23776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:06.949039 23776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:06.950280 23776 hybrid_clock.cc:648] HybridClock initialized: now 1790246646950242 us; error 46 us; skew 500 ppm
Sep 24 10:44:06 dist-test-slave-j38h krb5kdc[23763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246646, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:06.953433 23776 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:44:06.954378 23776 webserver.cc:460] Webserver started at http://127.2.8.62:33149/ using document root <none> and password file <none>
I20260924 10:44:06.954985 23776 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:06.955086 23776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:06.955336 23776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:06.957543 23776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "0e6ef22fa4e64c659c63e8fbe9fdcf41"
format_stamp: "Formatted at 2026-09-24 10:44:06 on dist-test-slave-j38h"
server_key: "65e0e7faad1b53002462f95ca1837760"
server_key_iv: "89c1560091aaf71cca87e29d140f6144"
server_key_version: "encryptionkey@0"
I20260924 10:44:06.958163 23776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "0e6ef22fa4e64c659c63e8fbe9fdcf41"
format_stamp: "Formatted at 2026-09-24 10:44:06 on dist-test-slave-j38h"
server_key: "65e0e7faad1b53002462f95ca1837760"
server_key_iv: "89c1560091aaf71cca87e29d140f6144"
server_key_version: "encryptionkey@0"
I20260924 10:44:06.962160 23776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 10:44:06.964784 23791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:06.966130 23776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:06.966288 23776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "0e6ef22fa4e64c659c63e8fbe9fdcf41"
format_stamp: "Formatted at 2026-09-24 10:44:06 on dist-test-slave-j38h"
server_key: "65e0e7faad1b53002462f95ca1837760"
server_key_iv: "89c1560091aaf71cca87e29d140f6144"
server_key_version: "encryptionkey@0"
I20260924 10:44:06.966421 23776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:07.011618 23776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:07.014705 23776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:07.014931 23776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:07.022050 23776 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:42889
I20260924 10:44:07.022078 23843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:42889 every 8 connection(s)
I20260924 10:44:07.022989 23776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:07.025697 23844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:07.029223 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 23776
I20260924 10:44:07.029376 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:44:07.029673 2080 external_mini_cluster.cc:1428] Setting key 4fcacdd08731792a0e48d3768ba95d4a
I20260924 10:44:07.031677 23844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41: Bootstrap starting.
I20260924 10:44:07.033948 23844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:07.034687 23844 log.cc:824] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:07.036551 23844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41: No bootstrap required, opened a new log
I20260924 10:44:07.038868 23847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:07.031126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44277 (local address 127.2.8.62:42889)
0924 10:44:07.031562 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:07.031577 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:07.031612 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:44:07.032281 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:07.032298 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:07.032646 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:07.032979 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:07.033440 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:07.034486 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:07.035441 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:07.035914 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:07.036084 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:07.037028 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:07.037034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:07.037051 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:07.037447 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:07.037502 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:07.037524 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:07.037661 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:07.037847 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 10:44:07.037986 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":131,"threads_started":1}
I20260924 10:44:07.039915 23844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6ef22fa4e64c659c63e8fbe9fdcf41" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42889 } }
I20260924 10:44:07.040194 23844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:07.040275 23844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e6ef22fa4e64c659c63e8fbe9fdcf41, State: Initialized, Role: FOLLOWER
I20260924 10:44:07.040684 23844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6ef22fa4e64c659c63e8fbe9fdcf41" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42889 } }
I20260924 10:44:07.040833 23844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:07.040906 23844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:07.041034 23844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:07.042194 23844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6ef22fa4e64c659c63e8fbe9fdcf41" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42889 } }
I20260924 10:44:07.042340 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:44:07.042558 23844 leader_election.cc:302] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e6ef22fa4e64c659c63e8fbe9fdcf41; no voters:
I20260924 10:44:07.042886 23844 leader_election.cc:288] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:07.042965 23849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:07.043205 23849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [term 1 LEADER]: Becoming Leader. State: Replica: 0e6ef22fa4e64c659c63e8fbe9fdcf41, State: Running, Role: LEADER
I20260924 10:44:07.043560 23849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6ef22fa4e64c659c63e8fbe9fdcf41" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42889 } }
I20260924 10:44:07.044221 23844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:07.045282 23850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e6ef22fa4e64c659c63e8fbe9fdcf41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6ef22fa4e64c659c63e8fbe9fdcf41" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42889 } } }
I20260924 10:44:07.045423 23850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:07.045846 23851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e6ef22fa4e64c659c63e8fbe9fdcf41. Latest consensus state: current_term: 1 leader_uuid: "0e6ef22fa4e64c659c63e8fbe9fdcf41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e6ef22fa4e64c659c63e8fbe9fdcf41" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42889 } } }
I20260924 10:44:07.046000 23851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41 [sys.catalog]: This master's current role is: LEADER
W20260924 10:44:07.049333 23869 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:44:07.049465 23869 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:44:07.049558 23863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:07.049587 23847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:07.044613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58005 (local address 127.2.8.62:42889)
0924 10:44:07.044705 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:07.044709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:07.045228 (+ 519us) server_negotiation.cc:407] Connection header received
0924 10:44:07.045446 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:07.045449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:07.045500 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:07.045560 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:07.046046 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:07.046816 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:07.047810 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:07.048059 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:07.048177 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:07.049072 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:07.049075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:07.049078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:07.049098 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:07.049121 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:07.049133 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:07.049234 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:07.049370 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 10:44:07.049405 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 10:44:07.050702 23863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:07.056491 23863 catalog_manager.cc:1347] Generated new cluster ID: 0db403f022ca43259a851fa83d42e908
I20260924 10:44:07.056566 23863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:07.067378 23863 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:07.068655 23863 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:07.073540 23863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e6ef22fa4e64c659c63e8fbe9fdcf41: Generated new TSK 0
I20260924 10:44:07.074357 23863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:44:07.086778 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 23776
W20260924 10:44:07.092226 23761 connection.cc:535] client connection to 127.2.8.62:42889 recv error: Network error: failed to read from TLS socket (remote: 127.2.8.62:42889): Connection reset by peer (error 104)
2026-09-24T10:44:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:07 dist-test-slave-j38h krb5kdc[23875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:07 dist-test-slave-j38h krb5kdc[23875](info): set up 2 sockets
Sep 24 10:44:07 dist-test-slave-j38h krb5kdc[23875](info): commencing operation
krb5kdc: starting...
W20260924 10:44:08.133307 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:44:08 dist-test-slave-j38h krb5kdc[23875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:44:08.259876 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:45765
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:35465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:45765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:08.373456 23888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:08.373796 23888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:08.373890 23888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:08.377887 23888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:08.377996 23888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:08.378041 23888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:08.378108 23888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:08.378152 23888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:08.383570 23888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:35465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:45765
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:45765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:08.384688 23888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:08.385756 23888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:08.394559 23896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:08.394680 23894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:08.394636 23888 server_base.cc:1034] running on GCE node
W20260924 10:44:08.394657 23893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:08.395346 23888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:08.396450 23888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:08.397619 23888 hybrid_clock.cc:648] HybridClock initialized: now 1790246648397597 us; error 34 us; skew 500 ppm
Sep 24 10:44:08 dist-test-slave-j38h krb5kdc[23875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246648, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:08.403964 23888 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:44:08.404850 23888 webserver.cc:460] Webserver started at http://127.2.8.62:41769/ using document root <none> and password file <none>
I20260924 10:44:08.405465 23888 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:08.405526 23888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:08.405699 23888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:08.407708 23888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "524d9184645844be8d99d41379422099"
format_stamp: "Formatted at 2026-09-24 10:44:08 on dist-test-slave-j38h"
server_key: "1380f62fd5cf57b6a0f0edbd7911ef4d"
server_key_iv: "fc530e3fa71c3c6c8bfa9c8f4c94fc8c"
server_key_version: "encryptionkey@0"
I20260924 10:44:08.408274 23888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "524d9184645844be8d99d41379422099"
format_stamp: "Formatted at 2026-09-24 10:44:08 on dist-test-slave-j38h"
server_key: "1380f62fd5cf57b6a0f0edbd7911ef4d"
server_key_iv: "fc530e3fa71c3c6c8bfa9c8f4c94fc8c"
server_key_version: "encryptionkey@0"
I20260924 10:44:08.411929 23888 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 10:44:08.414289 23903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:08.415472 23888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:08.415596 23888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "524d9184645844be8d99d41379422099"
format_stamp: "Formatted at 2026-09-24 10:44:08 on dist-test-slave-j38h"
server_key: "1380f62fd5cf57b6a0f0edbd7911ef4d"
server_key_iv: "fc530e3fa71c3c6c8bfa9c8f4c94fc8c"
server_key_version: "encryptionkey@0"
I20260924 10:44:08.415692 23888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:08.420181 23888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:08.423156 23888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:08.423388 23888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:08.430863 23888 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:45765
I20260924 10:44:08.430889 23955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:45765 every 8 connection(s)
I20260924 10:44:08.431851 23888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:08.434644 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:08.436414 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 23888
I20260924 10:44:08.436579 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:44:08.436910 2080 external_mini_cluster.cc:1428] Setting key 39aadc05ffe57d9c8adac797533bc567
I20260924 10:44:08.441695 23956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099: Bootstrap starting.
I20260924 10:44:08.442749 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:08.438466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40987 (local address 127.2.8.62:45765)
0924 10:44:08.438902 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:08.438921 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:08.438964 (+ 43us) server_negotiation.cc:407] Connection header received
0924 10:44:08.439727 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:08.439757 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:08.440133 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:08.440522 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:08.441331 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:08.441338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:08.441353 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:08.441364 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:08.441434 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:08.441461 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:08.441599 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:08.441729 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 10:44:08.441884 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":129,"threads_started":1}
I20260924 10:44:08.444272 23956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:08.445144 23956 log.cc:824] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:08.445981 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:44:08.447363 23956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099: No bootstrap required, opened a new log
I20260924 10:44:08.449855 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:08.448195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53125 (local address 127.2.8.62:45765)
0924 10:44:08.448321 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:08.448326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:08.448520 (+ 194us) server_negotiation.cc:407] Connection header received
0924 10:44:08.448634 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:08.448637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:08.448664 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:08.448727 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:08.449447 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:08.449450 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:08.449452 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:08.449462 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:08.449487 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:08.449496 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:08.449563 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:08.449673 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 10:44:08.449702 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 10:44:08.451704 23956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524d9184645844be8d99d41379422099" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45765 } }
I20260924 10:44:08.451973 23956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:08.452029 23956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524d9184645844be8d99d41379422099, State: Initialized, Role: FOLLOWER
I20260924 10:44:08.452436 23956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524d9184645844be8d99d41379422099" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45765 } }
I20260924 10:44:08.452590 23956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:08.452659 23956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:08.452759 23956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:08.453742 23956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524d9184645844be8d99d41379422099" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45765 } }
I20260924 10:44:08.454098 23956 leader_election.cc:302] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 524d9184645844be8d99d41379422099; no voters:
I20260924 10:44:08.454411 23956 leader_election.cc:288] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:08.454559 23966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:08.454810 23966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [term 1 LEADER]: Becoming Leader. State: Replica: 524d9184645844be8d99d41379422099, State: Running, Role: LEADER
I20260924 10:44:08.455173 23966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524d9184645844be8d99d41379422099" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45765 } }
I20260924 10:44:08.455788 23956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:08.456951 23968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 524d9184645844be8d99d41379422099. Latest consensus state: current_term: 1 leader_uuid: "524d9184645844be8d99d41379422099" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524d9184645844be8d99d41379422099" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45765 } } }
I20260924 10:44:08.457121 23968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:08.457522 23975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:08.457515 23967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "524d9184645844be8d99d41379422099" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524d9184645844be8d99d41379422099" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 45765 } } }
I20260924 10:44:08.457727 23967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:08.461031 23975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:08.466161 23975 catalog_manager.cc:1347] Generated new cluster ID: 0cb76d3ac26741bbad615e579095164a
I20260924 10:44:08.466236 23975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:08.487697 23975 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:08.488088 23975 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:08.498056 23975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 524d9184645844be8d99d41379422099: Generated new TSK 0
I20260924 10:44:08.498804 23975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:44:08.501829 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 23888
2026-09-24T10:44:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:08 dist-test-slave-j38h krb5kdc[23987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:08 dist-test-slave-j38h krb5kdc[23987](info): set up 2 sockets
Sep 24 10:44:08 dist-test-slave-j38h krb5kdc[23987](info): commencing operation
krb5kdc: starting...
W20260924 10:44:09.537256 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:44:09 dist-test-slave-j38h krb5kdc[23987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:44:09.669273 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:38725
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:34923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:38725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:09.780681 24000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:09.780947 24000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:09.781052 24000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:09.785089 24000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:09.785141 24000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:09.785161 24000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:09.785177 24000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:09.785192 24000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:09.790743 24000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:34923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:38725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:38725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:09.791766 24000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:09.792719 24000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:09.801199 24006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:09.801199 24005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:09.801244 24000 server_base.cc:1034] running on GCE node
W20260924 10:44:09.801247 24008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:09.802017 24000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:09.803089 24000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:09.804291 24000 hybrid_clock.cc:648] HybridClock initialized: now 1790246649804276 us; error 30 us; skew 500 ppm
Sep 24 10:44:09 dist-test-slave-j38h krb5kdc[23987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246649, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:09.807102 24000 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:44:09.807914 24000 webserver.cc:460] Webserver started at http://127.2.8.62:34597/ using document root <none> and password file <none>
I20260924 10:44:09.808429 24000 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:09.808486 24000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:09.808646 24000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:09.810675 24000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "a25af589deaa4fa48fa570429584ac7d"
format_stamp: "Formatted at 2026-09-24 10:44:09 on dist-test-slave-j38h"
server_key: "67ea1ae55485dd8056b6eb4cc4f151a2"
server_key_iv: "320c7f900d7561be66c8b737fc03980a"
server_key_version: "encryptionkey@0"
I20260924 10:44:09.811172 24000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "a25af589deaa4fa48fa570429584ac7d"
format_stamp: "Formatted at 2026-09-24 10:44:09 on dist-test-slave-j38h"
server_key: "67ea1ae55485dd8056b6eb4cc4f151a2"
server_key_iv: "320c7f900d7561be66c8b737fc03980a"
server_key_version: "encryptionkey@0"
I20260924 10:44:09.814868 24000 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 10:44:09.817461 24015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:09.818712 24000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:09.818813 24000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "a25af589deaa4fa48fa570429584ac7d"
format_stamp: "Formatted at 2026-09-24 10:44:09 on dist-test-slave-j38h"
server_key: "67ea1ae55485dd8056b6eb4cc4f151a2"
server_key_iv: "320c7f900d7561be66c8b737fc03980a"
server_key_version: "encryptionkey@0"
I20260924 10:44:09.818948 24000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:09.842163 24000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:09.845395 24000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:09.845623 24000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:09.852831 24000 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:38725
I20260924 10:44:09.852857 24067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:38725 every 8 connection(s)
I20260924 10:44:09.853811 24000 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:09.855836 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 24000
I20260924 10:44:09.855993 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:44:09.856287 2080 external_mini_cluster.cc:1428] Setting key 4dc030cf7eaff7aa7c9cc166eedb7b88
I20260924 10:44:09.856832 24068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:09.862744 24068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d: Bootstrap starting.
Sep 24 10:44:09 dist-test-slave-j38h krb5kdc[23987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:44:09.864925 24068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:09.865676 24068 log.cc:824] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:09.868220 24068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d: No bootstrap required, opened a new log
I20260924 10:44:09.870654 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:09.857752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54677 (local address 127.2.8.62:38725)
0924 10:44:09.858215 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:09.858230 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:09.858265 (+ 35us) server_negotiation.cc:407] Connection header received
0924 10:44:09.858906 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:09.858930 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:09.859256 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:09.859587 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:09.860386 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:09.861453 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:09.862306 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:09.862774 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:09.862961 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:09.865472 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:09.865477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:09.865490 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:09.865516 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:09.867525 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:09.867945 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:09.867951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:09.867956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:09.868097 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:09.868380 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:09.868384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:09.868385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:09.868882 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:09.869090 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:09.869352 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 10:44:09.869647 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":144,"threads_started":1}
I20260924 10:44:09.872680 24068 raft_consensus.cc:348] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25af589deaa4fa48fa570429584ac7d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38725 } }
I20260924 10:44:09.872944 24068 raft_consensus.cc:374] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:09.873042 24068 raft_consensus.cc:729] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a25af589deaa4fa48fa570429584ac7d, State: Initialized, Role: FOLLOWER
I20260924 10:44:09.873543 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:44:09.873545 24068 consensus_queue.cc:260] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25af589deaa4fa48fa570429584ac7d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38725 } }
I20260924 10:44:09.873714 24068 raft_consensus.cc:388] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:09.873783 24068 raft_consensus.cc:482] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:09.873898 24068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:09.874846 24068 raft_consensus.cc:504] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25af589deaa4fa48fa570429584ac7d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38725 } }
I20260924 10:44:09.875197 24068 leader_election.cc:302] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a25af589deaa4fa48fa570429584ac7d; no voters:
I20260924 10:44:09.875551 24068 leader_election.cc:288] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:09.875645 24077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:09.875922 24077 raft_consensus.cc:686] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [term 1 LEADER]: Becoming Leader. State: Replica: a25af589deaa4fa48fa570429584ac7d, State: Running, Role: LEADER
I20260924 10:44:09.876215 24077 consensus_queue.cc:237] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25af589deaa4fa48fa570429584ac7d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38725 } }
I20260924 10:44:09.876785 24068 sys_catalog.cc:564] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:09.877810 24080 sys_catalog.cc:455] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a25af589deaa4fa48fa570429584ac7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25af589deaa4fa48fa570429584ac7d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38725 } } }
I20260924 10:44:09.877938 24080 sys_catalog.cc:458] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:09.878310 24079 sys_catalog.cc:455] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a25af589deaa4fa48fa570429584ac7d. Latest consensus state: current_term: 1 leader_uuid: "a25af589deaa4fa48fa570429584ac7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a25af589deaa4fa48fa570429584ac7d" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38725 } } }
I20260924 10:44:09.878456 24079 sys_catalog.cc:458] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:09.878818 24087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:09.881735 24087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:09.885771 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:09.875707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43107 (local address 127.2.8.62:38725)
0924 10:44:09.875829 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:09.875835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:09.877255 (+ 1420us) server_negotiation.cc:407] Connection header received
0924 10:44:09.877463 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:09.877466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:09.877519 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:09.877591 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:09.878521 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:09.879336 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:09.880299 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:09.880617 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:09.880713 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:09.882476 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:09.882480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:09.882483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:09.882507 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:09.884108 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:09.884540 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:09.884545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:09.884547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:09.884625 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:09.884897 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:09.884902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:09.884904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:09.885133 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:09.885265 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:09.885506 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 10:44:09.885621 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 10:44:09.887553 24087 catalog_manager.cc:1347] Generated new cluster ID: a23f4178ff064e60ac0711483855b289
I20260924 10:44:09.887622 24087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:09.907848 24087 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:09.909140 24087 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:09.916322 24087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a25af589deaa4fa48fa570429584ac7d: Generated new TSK 0
I20260924 10:44:09.917150 24087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:44:09.928196 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 24000
2026-09-24T10:44:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:09 dist-test-slave-j38h krb5kdc[24099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:09 dist-test-slave-j38h krb5kdc[24099](info): set up 2 sockets
Sep 24 10:44:09 dist-test-slave-j38h krb5kdc[24099](info): commencing operation
krb5kdc: starting...
W20260924 10:44:10.989305 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:44:11 dist-test-slave-j38h krb5kdc[24099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:44:11.112812 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:42717
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:42717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:11.226405 24112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:11.226739 24112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:11.226820 24112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:11.230872 24112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:11.230985 24112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:11.231024 24112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:11.231061 24112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:11.231097 24112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:11.236784 24112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:42717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:42717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:11.238021 24112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:11.239135 24112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:11.247927 24120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:11.247964 24118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:11.248102 24112 server_base.cc:1034] running on GCE node
W20260924 10:44:11.247934 24117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:11.248950 24112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:11.250169 24112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:11.251365 24112 hybrid_clock.cc:648] HybridClock initialized: now 1790246651251349 us; error 28 us; skew 500 ppm
Sep 24 10:44:11 dist-test-slave-j38h krb5kdc[24099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246651, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:11.254511 24112 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:44:11.255471 24112 webserver.cc:460] Webserver started at http://127.2.8.62:46873/ using document root <none> and password file <none>
I20260924 10:44:11.256049 24112 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:11.256145 24112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:11.256382 24112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:11.258502 24112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "ee9c7d054e1f4ff4a664471f3efc8826"
format_stamp: "Formatted at 2026-09-24 10:44:11 on dist-test-slave-j38h"
server_key: "d2d3f45c044e02c228b2a2ad425cd0c5"
server_key_iv: "4e63cc1a76a1bc10640c614bdaf2a1e9"
server_key_version: "encryptionkey@0"
I20260924 10:44:11.259127 24112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "ee9c7d054e1f4ff4a664471f3efc8826"
format_stamp: "Formatted at 2026-09-24 10:44:11 on dist-test-slave-j38h"
server_key: "d2d3f45c044e02c228b2a2ad425cd0c5"
server_key_iv: "4e63cc1a76a1bc10640c614bdaf2a1e9"
server_key_version: "encryptionkey@0"
I20260924 10:44:11.262848 24112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:44:11.265338 24127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:11.266637 24112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 10:44:11.266801 24112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "ee9c7d054e1f4ff4a664471f3efc8826"
format_stamp: "Formatted at 2026-09-24 10:44:11 on dist-test-slave-j38h"
server_key: "d2d3f45c044e02c228b2a2ad425cd0c5"
server_key_iv: "4e63cc1a76a1bc10640c614bdaf2a1e9"
server_key_version: "encryptionkey@0"
I20260924 10:44:11.266921 24112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:11.284224 24112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:11.287808 24112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:11.288035 24112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:11.295548 24112 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:42717
I20260924 10:44:11.295583 24179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:42717 every 8 connection(s)
I20260924 10:44:11.296545 24112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:11.299269 24180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:11.299602 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 24112
I20260924 10:44:11.299778 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:44:11.300115 2080 external_mini_cluster.cc:1428] Setting key f8f9de762e6428e8029888876876faef
I20260924 10:44:11.305632 24180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826: Bootstrap starting.
I20260924 10:44:11.307809 24180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:11.308682 24180 log.cc:824] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:11.309419 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:11.301569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33969 (local address 127.2.8.62:42717)
0924 10:44:11.301996 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:11.302010 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:11.302046 (+ 36us) server_negotiation.cc:407] Connection header received
0924 10:44:11.302723 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:11.302749 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:11.303083 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:11.303430 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:11.303855 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:11.304816 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:11.305803 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:11.306299 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:11.306482 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:11.307400 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:11.307407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:11.307431 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:11.307829 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:11.307890 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:11.307938 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:11.308053 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:11.308232 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 10:44:11.308463 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
I20260924 10:44:11.310842 24180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826: No bootstrap required, opened a new log
I20260924 10:44:11.312501 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:44:11.315508 24180 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9c7d054e1f4ff4a664471f3efc8826" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42717 } }
I20260924 10:44:11.315836 24180 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:11.315899 24180 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9c7d054e1f4ff4a664471f3efc8826, State: Initialized, Role: FOLLOWER
I20260924 10:44:11.316387 24180 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9c7d054e1f4ff4a664471f3efc8826" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42717 } }
I20260924 10:44:11.316553 24180 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:11.316622 24180 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:11.316720 24180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:11.317942 24180 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9c7d054e1f4ff4a664471f3efc8826" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42717 } }
I20260924 10:44:11.318338 24180 leader_election.cc:302] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee9c7d054e1f4ff4a664471f3efc8826; no voters:
I20260924 10:44:11.318694 24180 leader_election.cc:288] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:11.318804 24190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:11.319082 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:11.314839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50299 (local address 127.2.8.62:42717)
0924 10:44:11.314973 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:11.314977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:11.315147 (+ 170us) server_negotiation.cc:407] Connection header received
0924 10:44:11.315279 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:11.315281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:11.315321 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:11.315379 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:11.315711 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:11.316319 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:11.317441 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:11.317686 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:11.317768 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:11.318603 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:11.318606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:11.318608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:11.318619 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:11.318646 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:11.318656 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:11.318746 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:11.318879 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 10:44:11.318940 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 10:44:11.319224 24190 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [term 1 LEADER]: Becoming Leader. State: Replica: ee9c7d054e1f4ff4a664471f3efc8826, State: Running, Role: LEADER
I20260924 10:44:11.319557 24190 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9c7d054e1f4ff4a664471f3efc8826" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42717 } }
I20260924 10:44:11.320207 24180 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:11.320714 24190 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee9c7d054e1f4ff4a664471f3efc8826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9c7d054e1f4ff4a664471f3efc8826" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42717 } } }
I20260924 10:44:11.320829 24190 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:11.321117 24190 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee9c7d054e1f4ff4a664471f3efc8826. Latest consensus state: current_term: 1 leader_uuid: "ee9c7d054e1f4ff4a664471f3efc8826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee9c7d054e1f4ff4a664471f3efc8826" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 42717 } } }
I20260924 10:44:11.321226 24190 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:11.321542 24195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:11.324682 24195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:11.330253 24195 catalog_manager.cc:1347] Generated new cluster ID: 241f43e8fafa44c5bee6557b572c9a16
I20260924 10:44:11.330344 24195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:11.341984 24195 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:11.342926 24195 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:11.351411 24195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee9c7d054e1f4ff4a664471f3efc8826: Generated new TSK 0
I20260924 10:44:11.352144 24195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:44:11.362769 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 24112
2026-09-24T10:44:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:11 dist-test-slave-j38h krb5kdc[24211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:11 dist-test-slave-j38h krb5kdc[24211](info): set up 2 sockets
Sep 24 10:44:11 dist-test-slave-j38h krb5kdc[24211](info): commencing operation
krb5kdc: starting...
W20260924 10:44:12.429476 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:44:12 dist-test-slave-j38h krb5kdc[24211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:44:12.556651 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:38981
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:39347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:38981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:12.666599 24224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:12.666875 24224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:12.666917 24224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:12.671025 24224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:12.671094 24224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:12.671114 24224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:12.671131 24224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:12.671147 24224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:12.676537 24224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:39347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:38981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:38981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:12.677692 24224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:12.678786 24224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:12.687394 24232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:12.687394 24229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:12.687395 24230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:12.687685 24224 server_base.cc:1034] running on GCE node
I20260924 10:44:12.688237 24224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:12.689452 24224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:12.690652 24224 hybrid_clock.cc:648] HybridClock initialized: now 1790246652690613 us; error 53 us; skew 500 ppm
Sep 24 10:44:12 dist-test-slave-j38h krb5kdc[24211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246652, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:12.693655 24224 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:44:12.694697 24224 webserver.cc:460] Webserver started at http://127.2.8.62:46183/ using document root <none> and password file <none>
I20260924 10:44:12.695288 24224 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:12.695389 24224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:12.695633 24224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:12.697863 24224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "cd16804d3bba4af68306e301763393bd"
format_stamp: "Formatted at 2026-09-24 10:44:12 on dist-test-slave-j38h"
server_key: "d46d04249896ebee5237c01f010630c9"
server_key_iv: "c5e04adbb722d5b4703227f6123c003d"
server_key_version: "encryptionkey@0"
I20260924 10:44:12.698470 24224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "cd16804d3bba4af68306e301763393bd"
format_stamp: "Formatted at 2026-09-24 10:44:12 on dist-test-slave-j38h"
server_key: "d46d04249896ebee5237c01f010630c9"
server_key_iv: "c5e04adbb722d5b4703227f6123c003d"
server_key_version: "encryptionkey@0"
I20260924 10:44:12.702318 24224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 10:44:12.705144 24239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:12.706580 24224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 10:44:12.706737 24224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "cd16804d3bba4af68306e301763393bd"
format_stamp: "Formatted at 2026-09-24 10:44:12 on dist-test-slave-j38h"
server_key: "d46d04249896ebee5237c01f010630c9"
server_key_iv: "c5e04adbb722d5b4703227f6123c003d"
server_key_version: "encryptionkey@0"
I20260924 10:44:12.706857 24224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:12.711395 24224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:12.714368 24224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:12.714565 24224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:12.721227 24224 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:38981
I20260924 10:44:12.721252 24291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:38981 every 8 connection(s)
I20260924 10:44:12.722117 24224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:12.722524 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 24224
I20260924 10:44:12.722685 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:44:12.722926 2080 external_mini_cluster.cc:1428] Setting key fe472e0eb2bcc1c4781dea352b2c1ae3
I20260924 10:44:12.725029 24292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:12.727967 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:12.724259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41159 (local address 127.2.8.62:38981)
0924 10:44:12.724642 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:12.724657 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:12.724690 (+ 33us) server_negotiation.cc:407] Connection header received
0924 10:44:12.725441 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:12.725464 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:12.725738 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:12.726016 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:12.726826 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:12.726831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:12.726842 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:12.726851 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:12.726901 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:12.726918 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:12.727027 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:12.727129 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 10:44:12.727281 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":133,"threads_started":1}
I20260924 10:44:12.731014 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:44:12.731606 24292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd: Bootstrap starting.
I20260924 10:44:12.733980 24292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:12.734865 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:12.733086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37443 (local address 127.2.8.62:38981)
0924 10:44:12.733205 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:12.733208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:12.733433 (+ 225us) server_negotiation.cc:407] Connection header received
0924 10:44:12.733579 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:12.733584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:12.733623 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:12.733707 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:12.734480 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:12.734483 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:12.734485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 10:44:12.734494 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:12.734518 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 10:44:12.734527 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:12.734596 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:12.734706 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 10:44:12.734737 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 10:44:12.734975 24292 log.cc:824] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:12.736845 24292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd: No bootstrap required, opened a new log
I20260924 10:44:12.740288 24292 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd16804d3bba4af68306e301763393bd" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38981 } }
I20260924 10:44:12.740513 24292 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 10:44:12.740592 24292 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd16804d3bba4af68306e301763393bd, State: Initialized, Role: FOLLOWER
I20260924 10:44:12.741047 24292 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd16804d3bba4af68306e301763393bd" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38981 } }
I20260924 10:44:12.741195 24292 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 10:44:12.741262 24292 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 10:44:12.741365 24292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:12.742287 24292 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd16804d3bba4af68306e301763393bd" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38981 } }
I20260924 10:44:12.742650 24292 leader_election.cc:302] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd16804d3bba4af68306e301763393bd; no voters:
I20260924 10:44:12.742954 24292 leader_election.cc:288] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 10:44:12.743095 24302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:12.743309 24302 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [term 1 LEADER]: Becoming Leader. State: Replica: cd16804d3bba4af68306e301763393bd, State: Running, Role: LEADER
I20260924 10:44:12.743640 24302 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd16804d3bba4af68306e301763393bd" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38981 } }
I20260924 10:44:12.744184 24292 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:12.745306 24303 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd16804d3bba4af68306e301763393bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd16804d3bba4af68306e301763393bd" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38981 } } }
I20260924 10:44:12.745438 24303 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:12.745631 24304 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd16804d3bba4af68306e301763393bd. Latest consensus state: current_term: 1 leader_uuid: "cd16804d3bba4af68306e301763393bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd16804d3bba4af68306e301763393bd" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 38981 } } }
I20260924 10:44:12.745769 24304 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:12.745844 24311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:12.748920 24311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:12.754222 24311 catalog_manager.cc:1347] Generated new cluster ID: ecea30dfc259434e9a1279ddb372c444
I20260924 10:44:12.754289 24311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:12.763779 24311 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:12.764086 24311 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:12.768590 24311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd16804d3bba4af68306e301763393bd: Generated new TSK 0
I20260924 10:44:12.769259 24311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 10:44:12.777109 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 24224
2026-09-24T10:44:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1413 ms)
[----------] 18 tests from AuthTokenIssuingTest (25777 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:12 dist-test-slave-j38h krb5kdc[24323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:12 dist-test-slave-j38h krb5kdc[24323](info): set up 2 sockets
Sep 24 10:44:12 dist-test-slave-j38h krb5kdc[24323](info): commencing operation
krb5kdc: starting...
W20260924 10:44:13.805362 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:44:13 dist-test-slave-j38h krb5kdc[24323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:44:13.929102 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:43191
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:46139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:43191,127.2.8.61:42729,127.2.8.60:45837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:14.036895 24336 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:44:14.037191 24336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:14.037213 24336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:14.037251 24336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:14.041142 24336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:14.041193 24336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:14.041210 24336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:14.041226 24336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:14.041241 24336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:14.046245 24336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:46139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:43191,127.2.8.61:42729,127.2.8.60:45837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:43191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:14.047166 24336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:14.048084 24336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:44:14.056584 24336 server_base.cc:1034] running on GCE node
W20260924 10:44:14.056536 24341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:14.056551 24342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:14.056526 24344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:14.057379 24336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:14.058507 24336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:14.059695 24336 hybrid_clock.cc:648] HybridClock initialized: now 1790246654059663 us; error 48 us; skew 500 ppm
Sep 24 10:44:14 dist-test-slave-j38h krb5kdc[24323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246654, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:14.067524 24336 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:44:14.068480 24336 webserver.cc:460] Webserver started at http://127.2.8.62:43699/ using document root <none> and password file <none>
I20260924 10:44:14.069062 24336 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:14.069129 24336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:14.069324 24336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:14.071305 24336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3"
format_stamp: "Formatted at 2026-09-24 10:44:14 on dist-test-slave-j38h"
server_key: "7ca4943d7b9475ccd96cc75f00a4e429"
server_key_iv: "32d5c19f6def848ed5b9296da0a2aa44"
server_key_version: "encryptionkey@0"
I20260924 10:44:14.071810 24336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3"
format_stamp: "Formatted at 2026-09-24 10:44:14 on dist-test-slave-j38h"
server_key: "7ca4943d7b9475ccd96cc75f00a4e429"
server_key_iv: "32d5c19f6def848ed5b9296da0a2aa44"
server_key_version: "encryptionkey@0"
I20260924 10:44:14.075520 24336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 10:44:14.077795 24351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:14.078982 24336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260924 10:44:14.079077 24336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3"
format_stamp: "Formatted at 2026-09-24 10:44:14 on dist-test-slave-j38h"
server_key: "7ca4943d7b9475ccd96cc75f00a4e429"
server_key_iv: "32d5c19f6def848ed5b9296da0a2aa44"
server_key_version: "encryptionkey@0"
I20260924 10:44:14.079155 24336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:14.099723 24336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:14.102919 24336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:14.103068 24336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:14.109923 24336 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:43191
I20260924 10:44:14.110219 24403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:43191 every 8 connection(s)
I20260924 10:44:14.110878 24336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:14.113588 24404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:14.115031 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 24336
I20260924 10:44:14.115183 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:44:14.115466 2080 external_mini_cluster.cc:1428] Setting key 568ebe1751be5fe6f346ed752a8ece03
I20260924 10:44:14.117348 24404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.8.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.61@KRBTEST.COM" created.
Sep 24 10:44:14 dist-test-slave-j38h krb5kdc[24323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246654, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:44:14.132848 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.119429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43191 (local address 127.0.0.1:45876)
0924 10:44:14.119884 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.119934 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:14.120853 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:14.121702 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:14.121706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:14.122021 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:14.122270 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.122282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.123791 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.123793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.124693 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.124703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.125344 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.125396 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.125485 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.126090 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:14.126106 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:14.127712 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:14.130026 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.130034 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.130042 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.130258 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.130624 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.130626 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.130627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.130729 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.131506 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:14.131511 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:14.131629 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 10:44:14.131767 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 10:44:14.132025 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":136,"threads_started":1}
I20260924 10:44:14.132851 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.120300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.2.8.62:43191)
0924 10:44:14.120592 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.120607 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.120649 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:44:14.121232 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.121240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.121491 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.121700 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.122420 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.123685 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.124813 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.125243 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.125355 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.127861 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:14.127870 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:14.127884 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:14.127913 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:14.129911 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.130370 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.130379 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.130383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.130524 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.130824 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.130830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.130835 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.131386 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:14.131540 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:14.131883 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 10:44:14.132130 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
I20260924 10:44:14.135267 24404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:44:14.135955 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.135690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:42729 (local address 127.0.0.1:51600)
0924 10:44:14.135803 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.135892 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 10:44:14.136155 24352 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.8.61:42729: connect: Connection refused (error 111)
I20260924 10:44:14.136919 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.136732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:42729 (local address 127.0.0.1:51604)
0924 10:44:14.136833 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.136874 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 10:44:14.137109 24404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.61:42729: Network error: Client connection negotiation failed: client connection to 127.2.8.61:42729: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.61.keytab.
Entry for principal kudu/127.2.8.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.61.keytab.
I20260924 10:44:14.143618 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.61:42729
--webserver_interface=127.2.8.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.61.keytab
--principal=kudu/127.2.8.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:46139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:43191,127.2.8.61:42729,127.2.8.60:45837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:44:14.186384 24404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } attempt: 1
I20260924 10:44:14.187220 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.186930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:42729 (local address 127.0.0.1:51614)
0924 10:44:14.187073 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.187142 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 10:44:14.188069 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.187863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:42729 (local address 127.0.0.1:51618)
0924 10:44:14.187971 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.188015 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:44:14.188213 24404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.61:42729: Network error: Client connection negotiation failed: client connection to 127.2.8.61:42729: connect: Connection refused (error 111)
W20260924 10:44:14.251842 24411 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:44:14.252123 24411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:14.252146 24411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:14.252183 24411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:14.256222 24411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:14.256281 24411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:14.256300 24411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:14.256317 24411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:14.256332 24411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:14.256407 24404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } attempt: 2
I20260924 10:44:14.257249 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.256953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:42729 (local address 127.0.0.1:51624)
0924 10:44:14.257116 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.257176 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 10:44:14.258005 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.257816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:42729 (local address 127.0.0.1:51636)
0924 10:44:14.257907 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.257954 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:42729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 10:44:14.258148 24404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.61:42729: Network error: Client connection negotiation failed: client connection to 127.2.8.61:42729: connect: Connection refused (error 111)
I20260924 10:44:14.262168 24411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:46139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:43191,127.2.8.61:42729,127.2.8.60:45837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.61.keytab
--principal=kudu/127.2.8.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.61:42729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.8.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:14.263150 24411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:14.264096 24411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:14.272578 24417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:14.272578 24416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:14.272612 24411 server_base.cc:1034] running on GCE node
W20260924 10:44:14.272617 24419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:14.273341 24411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:14.274405 24411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:14.275583 24411 hybrid_clock.cc:648] HybridClock initialized: now 1790246654275553 us; error 43 us; skew 500 ppm
Sep 24 10:44:14 dist-test-slave-j38h krb5kdc[24323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246654, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:14.278203 24411 init.cc:377] Logged in from keytab as kudu/127.2.8.61@KRBTEST.COM (short username kudu)
I20260924 10:44:14.279093 24411 webserver.cc:460] Webserver started at http://127.2.8.61:40481/ using document root <none> and password file <none>
I20260924 10:44:14.279631 24411 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:14.279718 24411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:14.279935 24411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:14.281991 24411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/data/instance:
uuid: "90a681830cd049c7950912b3fa7caaf6"
format_stamp: "Formatted at 2026-09-24 10:44:14 on dist-test-slave-j38h"
server_key: "62203e676c58fb1754afa4df54abe8c0"
server_key_iv: "d46069265d1d8f6735186d26bf483cb8"
server_key_version: "encryptionkey@0"
I20260924 10:44:14.282567 24411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/wal/instance:
uuid: "90a681830cd049c7950912b3fa7caaf6"
format_stamp: "Formatted at 2026-09-24 10:44:14 on dist-test-slave-j38h"
server_key: "62203e676c58fb1754afa4df54abe8c0"
server_key_iv: "d46069265d1d8f6735186d26bf483cb8"
server_key_version: "encryptionkey@0"
I20260924 10:44:14.286121 24411 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 10:44:14.288357 24426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:14.289635 24411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:14.289767 24411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/wal
uuid: "90a681830cd049c7950912b3fa7caaf6"
format_stamp: "Formatted at 2026-09-24 10:44:14 on dist-test-slave-j38h"
server_key: "62203e676c58fb1754afa4df54abe8c0"
server_key_iv: "d46069265d1d8f6735186d26bf483cb8"
server_key_version: "encryptionkey@0"
I20260924 10:44:14.289875 24411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:14.306938 24411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:14.309756 24411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:14.309955 24411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:14.316632 24411 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.61:42729
I20260924 10:44:14.316895 24478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.61:42729 every 8 connection(s)
I20260924 10:44:14.317622 24411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/data/info.pb
I20260924 10:44:14.319584 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 24411
I20260924 10:44:14.319684 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-1/wal/instance
I20260924 10:44:14.320003 2080 external_mini_cluster.cc:1428] Setting key 480a144d4672d13d7e858ef57e81c2ea
I20260924 10:44:14.320441 24479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:14.324179 24479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.8.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.60@KRBTEST.COM" created.
Sep 24 10:44:14 dist-test-slave-j38h krb5kdc[24323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246654, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.61@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:44:14.337725 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.326104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.2.8.62:43191)
0924 10:44:14.326261 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.326266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.327084 (+ 818us) server_negotiation.cc:407] Connection header received
0924 10:44:14.328099 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.328104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.328164 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.328247 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.329498 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.330309 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.331429 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.331687 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.331799 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.334174 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:14.334178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:14.334180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:14.334206 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:14.335650 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.336152 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.336156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.336158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.336240 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.336582 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.336587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.336589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.336769 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:14.336876 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:14.337431 (+ 555us) server_negotiation.cc:299] Negotiation successful
0924 10:44:14.337552 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 10:44:14.338601 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.326484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43191 (local address 127.0.0.1:45890)
0924 10:44:14.326936 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.326987 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:14.327862 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:14.328406 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:14.328412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:14.328745 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:14.329346 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.329360 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.330450 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.330453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.331298 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.331308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.331797 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.331851 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.332014 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.332601 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:14.332616 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:14.334010 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:14.335778 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.335784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.335798 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.336049 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.336340 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.336343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.336345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.336485 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.336875 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:14.336882 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:14.337178 (+ 296us) client_negotiation.cc:769] Sending connection context
0924 10:44:14.337363 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 10:44:14.337677 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":136,"threads_started":1}
I20260924 10:44:14.340113 24479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:44:14 dist-test-slave-j38h krb5kdc[24323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246654, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.61@KRBTEST.COM for kudu/127.2.8.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.60.keytab.
Entry for principal kudu/127.2.8.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.60.keytab.
I20260924 10:44:14.346410 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.60:45837
--webserver_interface=127.2.8.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.60.keytab
--principal=kudu/127.2.8.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:46139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:43191,127.2.8.61:42729,127.2.8.60:45837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:44:14.349321 24404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } attempt: 3
I20260924 10:44:14.349397 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.340574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:42729 (local address 127.0.0.1:51640)
0924 10:44:14.340701 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.340718 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:14.340793 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:14.341499 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:14.341501 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:14.341678 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:14.341753 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.341758 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.342740 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.342743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.343262 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.343275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.343848 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.343862 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.343953 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.344429 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:14.344444 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:14.345734 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:14.347706 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.347709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.347711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.347902 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.348187 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.348189 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.348191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.348269 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.348788 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:14.348790 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:14.348895 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 10:44:14.349087 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 10:44:14.349208 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 10:44:14.349807 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.340715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.2.8.61:42729)
0924 10:44:14.341098 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.341116 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.341134 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:44:14.341165 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.341189 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.341381 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.341523 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.341887 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.342624 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.343384 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.343736 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.343856 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.345865 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:14.345877 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:14.345895 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:14.345921 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:14.347569 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.348007 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.348013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.348014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.348092 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.348366 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.348369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.348371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.348636 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:14.348768 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:14.349181 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 10:44:14.349354 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":242,"thread_start_us":111,"threads_started":1}
I20260924 10:44:14.353266 24479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:44:14.354216 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.353772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45837 (local address 127.0.0.1:41268)
0924 10:44:14.353995 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.354104 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260924 10:44:14.354468 24428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
I20260924 10:44:14.355382 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.355141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45837 (local address 127.0.0.1:41272)
0924 10:44:14.355253 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.355305 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:44:14.355957 24479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:45837: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
Sep 24 10:44:14 dist-test-slave-j38h krb5kdc[24323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246654, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for kudu/127.2.8.61@KRBTEST.COM
I20260924 10:44:14.360337 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.349751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:42729 (local address 127.0.0.1:51648)
0924 10:44:14.349922 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.349935 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:14.350006 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:14.350647 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:14.350651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:14.350853 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:14.350977 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.350993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.351851 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.351857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.353067 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.353079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.353695 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.353714 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.353890 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.355316 (+ 1426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:14.355332 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:14.357015 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:14.358826 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.358829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.358833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.359078 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.359360 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.359363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.359365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.359454 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.359826 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:14.359829 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:14.359928 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 10:44:14.360044 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 10:44:14.360179 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 10:44:14.360337 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.350234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51648 (local address 127.2.8.61:42729)
0924 10:44:14.350384 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.350390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.350408 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:44:14.350447 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.350452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.350503 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.350593 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.351157 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.351710 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.353212 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.353570 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.353709 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.357149 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:14.357152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:14.357154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:14.357176 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:14.358702 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.359177 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.359179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.359181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.359260 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.359541 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.359544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.359545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.359717 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:14.359813 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:14.360096 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 10:44:14.360189 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:44:14.361851 24404 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:44:14.362576 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.362336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45837 (local address 127.0.0.1:41284)
0924 10:44:14.362455 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.362506 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:44:14.363358 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.363085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45837 (local address 127.0.0.1:41296)
0924 10:44:14.363236 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.363287 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260924 10:44:14.363517 24404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:45837: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
I20260924 10:44:14.394673 24404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } attempt: 1
I20260924 10:44:14.395541 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.395185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45837 (local address 127.0.0.1:41312)
0924 10:44:14.395367 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.395446 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 10:44:14.396327 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.396079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45837 (local address 127.0.0.1:41324)
0924 10:44:14.396207 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.396263 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 10:44:14.396474 24404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:45837: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
I20260924 10:44:14.405264 24479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } attempt: 1
I20260924 10:44:14.405998 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.405734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45837 (local address 127.0.0.1:41334)
0924 10:44:14.405875 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.405935 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 10:44:14.406739 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.406501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45837 (local address 127.0.0.1:41346)
0924 10:44:14.406610 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.406660 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 10:44:14.406906 24479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:45837: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
W20260924 10:44:14.460290 24485 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:44:14.460644 24485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:14.460718 24485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:14.460803 24485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:14.464919 24485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:14.465037 24485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:14.465080 24485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:14.465114 24485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:14.465149 24485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:14.470796 24485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:46139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:43191,127.2.8.61:42729,127.2.8.60:45837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.60.keytab
--principal=kudu/127.2.8.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.60:45837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.8.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:14.471635 24404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } attempt: 2
I20260924 10:44:14.472082 24485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:14.472541 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.472209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45837 (local address 127.0.0.1:41358)
0924 10:44:14.472376 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.472452 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 10:44:14.473347 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.473123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45837 (local address 127.0.0.1:41360)
0924 10:44:14.473231 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.473284 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:44:14.473476 24404 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:45837: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
I20260924 10:44:14.473570 24485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:44:14.475061 24479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } attempt: 2
I20260924 10:44:14.475711 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.475465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45837 (local address 127.0.0.1:41372)
0924 10:44:14.475585 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.475647 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 10:44:14.476365 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.476158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45837 (local address 127.0.0.1:41378)
0924 10:44:14.476259 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.476306 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 10:44:14.476495 24479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:45837: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45837: connect: Connection refused (error 111)
W20260924 10:44:14.482034 24492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:14.482043 24491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:14.482160 24494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:14.482290 24485 server_base.cc:1034] running on GCE node
I20260924 10:44:14.482936 24485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:14.483984 24485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:14.485165 24485 hybrid_clock.cc:648] HybridClock initialized: now 1790246654485152 us; error 31 us; skew 500 ppm
Sep 24 10:44:14 dist-test-slave-j38h krb5kdc[24323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246654, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:14.487919 24485 init.cc:377] Logged in from keytab as kudu/127.2.8.60@KRBTEST.COM (short username kudu)
I20260924 10:44:14.488812 24485 webserver.cc:460] Webserver started at http://127.2.8.60:33091/ using document root <none> and password file <none>
I20260924 10:44:14.489409 24485 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:14.489471 24485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:14.489635 24485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:14.491612 24485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/data/instance:
uuid: "6bc61198e44645f682ad42f1dea2a774"
format_stamp: "Formatted at 2026-09-24 10:44:14 on dist-test-slave-j38h"
server_key: "e8f23d10d12d51f6d96bfce6b5e86661"
server_key_iv: "9d1b7322bbc97046c8199d666847fa41"
server_key_version: "encryptionkey@0"
I20260924 10:44:14.492111 24485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/wal/instance:
uuid: "6bc61198e44645f682ad42f1dea2a774"
format_stamp: "Formatted at 2026-09-24 10:44:14 on dist-test-slave-j38h"
server_key: "e8f23d10d12d51f6d96bfce6b5e86661"
server_key_iv: "9d1b7322bbc97046c8199d666847fa41"
server_key_version: "encryptionkey@0"
I20260924 10:44:14.495903 24485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 10:44:14.498327 24501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:14.499517 24485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 10:44:14.499612 24485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/wal
uuid: "6bc61198e44645f682ad42f1dea2a774"
format_stamp: "Formatted at 2026-09-24 10:44:14 on dist-test-slave-j38h"
server_key: "e8f23d10d12d51f6d96bfce6b5e86661"
server_key_iv: "9d1b7322bbc97046c8199d666847fa41"
server_key_version: "encryptionkey@0"
I20260924 10:44:14.499693 24485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:14.515386 24485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:14.518700 24485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:14.518863 24485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:14.525928 24485 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.60:45837
I20260924 10:44:14.526237 24553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.60:45837 every 8 connection(s)
I20260924 10:44:14.526922 24485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/data/info.pb
I20260924 10:44:14.529690 24554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:14.533219 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 24485
I20260924 10:44:14.533346 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790246374820021-2080-0/minicluster-data/master-2/wal/instance
I20260924 10:44:14.533646 2080 external_mini_cluster.cc:1428] Setting key c2d8173afb077bdcf341d6cc9fc24c4b
I20260924 10:44:14.533525 24554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:44:14 dist-test-slave-j38h krb5kdc[24323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246654, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.60@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
Sep 24 10:44:14 dist-test-slave-j38h krb5kdc[24323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.60@KRBTEST.COM
I20260924 10:44:14.546298 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.535518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.2.8.62:43191)
0924 10:44:14.535663 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.535667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.535950 (+ 283us) server_negotiation.cc:407] Connection header received
0924 10:44:14.536480 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.536483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.536526 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.536596 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.537521 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.538013 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.539108 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.539457 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.539534 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.542170 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:14.542175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:14.542178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:14.542208 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:14.543916 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.544498 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.544501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.544503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.544581 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.544936 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.544944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.544948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.545234 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:14.545408 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:14.545991 (+ 583us) server_negotiation.cc:299] Negotiation successful
0924 10:44:14.546122 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:44:14.547216 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.535430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:43191 (local address 127.0.0.1:45902)
0924 10:44:14.535801 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.535850 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:14.536276 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:14.536663 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:14.536668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:14.537120 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:14.537354 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.537367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.538116 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.538119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.538996 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.539004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.539538 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.539581 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.539762 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.540417 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:14.540432 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:14.541957 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:14.544027 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.544036 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.544045 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.544380 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.544671 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.544675 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.544679 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.544835 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.545404 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:14.545408 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:14.545713 (+ 305us) client_negotiation.cc:769] Sending connection context
0924 10:44:14.545906 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 10:44:14.546216 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"spinlock_wait_cycles":2432,"thread_start_us":126,"threads_started":1}
I20260924 10:44:14.547216 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.534939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.2.8.60:45837)
0924 10:44:14.535338 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.535352 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.535384 (+ 32us) server_negotiation.cc:407] Connection header received
0924 10:44:14.536276 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.536299 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.536526 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.536685 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.537480 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.538489 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.539434 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.539903 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.540047 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.542732 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:14.542739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:14.542754 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:14.542780 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:14.544612 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.545145 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.545150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.545151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.545215 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.545533 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.545535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.545537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.545801 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:14.546001 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:14.546265 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 10:44:14.546395 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"spinlock_wait_cycles":768,"thread_start_us":129,"threads_started":1}
I20260924 10:44:14.548658 24554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:44:14 dist-test-slave-j38h krb5kdc[24323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246654, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.60@KRBTEST.COM for kudu/127.2.8.61@KRBTEST.COM
Sep 24 10:44:14 dist-test-slave-j38h krb5kdc[24323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:44:14.557169 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.549281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:42729 (local address 127.0.0.1:51664)
0924 10:44:14.549760 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.549774 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:14.549857 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:14.550088 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:14.550090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:14.550257 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:14.550341 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.550346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.551061 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.551068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.551577 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.551584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.552009 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.552023 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.552117 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.552542 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:14.552552 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:14.553763 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:14.555740 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.555742 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.555743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.555937 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.556225 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.556227 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.556228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.556296 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.556642 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:14.556647 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:14.556745 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 10:44:14.556857 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 10:44:14.557011 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409}
I20260924 10:44:14.557238 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.549392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.2.8.61:42729)
0924 10:44:14.549515 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.549519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.549827 (+ 308us) server_negotiation.cc:407] Connection header received
0924 10:44:14.549936 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.549939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.549991 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.550078 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.550463 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.550956 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.551701 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.551915 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.552021 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.553902 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:14.553905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:14.553907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:14.553928 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:14.555629 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.556042 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.556048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.556050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.556134 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.556373 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.556375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.556376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.556536 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:14.556609 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:14.556925 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 10:44:14.557085 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:44:14.558029 24554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:44:14.558212 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.549645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.2.8.62:43191)
0924 10:44:14.549760 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.549766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.549782 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:44:14.549889 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.549892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.549931 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.550020 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.550582 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.551055 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.551802 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.552025 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.552124 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.554532 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:14.554537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:14.554541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:14.554567 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:14.556140 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.556693 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.556695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.556697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.556765 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.557084 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.557087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.557088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.557273 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:14.557564 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:14.557908 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 10:44:14.558032 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 10:44:14 dist-test-slave-j38h krb5kdc[24323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246654, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.60@KRBTEST.COM for kudu/127.2.8.60@KRBTEST.COM
Sep 24 10:44:14 dist-test-slave-j38h krb5kdc[24323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.61@KRBTEST.COM
I20260924 10:44:14.566502 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.558491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45837 (local address 127.0.0.1:41410)
0924 10:44:14.558651 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.558668 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:14.558747 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:14.559001 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:14.559004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:14.559154 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:14.559242 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.559248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.560264 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.560266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.561057 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.561067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.561551 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.561564 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.561637 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.562096 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:14.562107 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:14.563295 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:14.565072 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.565076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.565080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.565325 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.565622 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.565626 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.565629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.565726 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.566149 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:14.566151 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:14.566221 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 10:44:14.566299 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 10:44:14.566389 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 10:44:14.566685 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.558553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.2.8.60:45837)
0924 10:44:14.558638 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.558641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.558710 (+ 69us) server_negotiation.cc:407] Connection header received
0924 10:44:14.558824 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.558826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.558884 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.558967 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.559359 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.560153 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.561185 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.561456 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.561566 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.563427 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:14.563431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:14.563436 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:14.563472 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:14.564937 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.565416 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.565420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.565422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.565525 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.565835 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.565838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.565839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.566055 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:14.566152 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:14.566379 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 10:44:14.566518 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 10:44:14.567644 24479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } attempt: 3
I20260924 10:44:14.568702 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.559410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.2.8.61:42729)
0924 10:44:14.559518 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.559522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.559541 (+ 19us) server_negotiation.cc:407] Connection header received
0924 10:44:14.559643 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.559645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.559677 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.559737 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.560452 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.561181 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.561904 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.562113 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.562187 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.564746 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:14.564753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:14.564759 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:14.564786 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:14.566507 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.566973 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.566978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.566983 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.567075 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.567356 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.567358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.567359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.567594 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:14.567829 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:14.568471 (+ 642us) server_negotiation.cc:299] Negotiation successful
0924 10:44:14.568596 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 10:44:14.570735 24554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774: Bootstrap starting.
Sep 24 10:44:14 dist-test-slave-j38h krb5kdc[24323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246654, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.61@KRBTEST.COM for kudu/127.2.8.60@KRBTEST.COM
I20260924 10:44:14.572666 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:44:14.572753 24404 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } attempt: 3
I20260924 10:44:14.573786 24554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:14.574734 24554 log.cc:824] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:14.574673 24378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45912
I20260924 10:44:14.576673 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.568022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45837 (local address 127.0.0.1:41418)
0924 10:44:14.568192 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.568206 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:14.568286 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:14.568526 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:14.568529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:14.568712 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:14.568802 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.568809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.569913 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.569917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.570550 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.570559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.571078 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.571092 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.571171 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.571596 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:14.571609 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:14.573251 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:14.575138 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.575144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.575149 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.575362 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.575685 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.575690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.575692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.575776 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.576183 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:14.576187 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:14.576276 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 10:44:14.576357 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:44:14.576507 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 10:44:14.576967 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.568093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.2.8.60:45837)
0924 10:44:14.568194 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.568199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.568250 (+ 51us) server_negotiation.cc:407] Connection header received
0924 10:44:14.568387 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.568390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.568426 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.568908 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.569044 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.569773 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.570661 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.570963 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.571358 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.573385 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:14.573389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:14.573391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:14.573413 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:14.575013 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.575510 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.575514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.575516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.575594 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.575884 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.575887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.575888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.576079 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:14.576605 (+ 526us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:14.576673 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 10:44:14.576783 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:44:14.578725 24554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774: No bootstrap required, opened a new log
I20260924 10:44:14.579859 24479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6: Bootstrap starting.
Sep 24 10:44:14 dist-test-slave-j38h krb5kdc[24323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246654, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for kudu/127.2.8.60@KRBTEST.COM
I20260924 10:44:14.582440 24479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:14.583035 24554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } }
I20260924 10:44:14.583278 24554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc61198e44645f682ad42f1dea2a774, State: Initialized, Role: FOLLOWER
I20260924 10:44:14.583389 24479 log.cc:824] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:14.583649 24554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } }
I20260924 10:44:14.584106 24562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } } }
I20260924 10:44:14.584240 24562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:14.584968 24554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:14.585651 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.576129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45837 (local address 127.0.0.1:41422)
0924 10:44:14.576260 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:14.576276 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:14.576351 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:14.576896 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:14.576902 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:14.577119 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:14.577227 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.577234 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.578294 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.578298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.579060 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:14.579070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.579645 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.579660 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:14.579737 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.580215 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:14.580229 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:14.581754 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:14.583991 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.583994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.583996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.584240 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.584580 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:14.584583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:14.584587 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:14.584708 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:14.585169 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:14.585172 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:14.585264 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 10:44:14.585359 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 10:44:14.585490 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 10:44:14.586376 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.576353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.2.8.60:45837)
0924 10:44:14.576634 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.576645 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.576660 (+ 15us) server_negotiation.cc:407] Connection header received
0924 10:44:14.576699 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.576705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.576749 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.577342 (+ 593us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.577454 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.578174 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.579185 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.579532 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.579637 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.581936 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:14.581941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:14.581943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:14.581967 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:14.583849 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.584366 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.584369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.584371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.584470 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.584816 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.584819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.584821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.585053 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:14.585968 (+ 915us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:14.586047 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 10:44:14.586198 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":120,"threads_started":1}
I20260924 10:44:14.586751 24479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6: No bootstrap required, opened a new log
I20260924 10:44:14.589534 24404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3: Bootstrap starting.
I20260924 10:44:14.590895 24479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } }
I20260924 10:44:14.591127 24479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a681830cd049c7950912b3fa7caaf6, State: Initialized, Role: FOLLOWER
I20260924 10:44:14.591460 24479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } }
I20260924 10:44:14.592082 24404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:14.589078 24485 master_runner.cc:186] Error getting master registration for 127.2.8.62:43191: OK, instance_id { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" instance_seqno: 1790246654108053 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:44:14.592242 24485 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:44:14.592787 24479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:14.592962 24404 log.cc:824] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3: Log is configured to *not* fsync() on all Append() calls
W20260924 10:44:14.593106 24573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:44:14.593180 24573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:44:14.595031 24404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3: No bootstrap required, opened a new log
I20260924 10:44:14.598109 24404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } }
I20260924 10:44:14.598337 24404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b6f435e26fa4578a69b1a16b2bcf3b3, State: Initialized, Role: FOLLOWER
I20260924 10:44:14.598649 24404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } }
I20260924 10:44:14.599200 24588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } } }
I20260924 10:44:14.599395 24588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:14.599901 24404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:14.601256 24378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 10:44:14.601387 24378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:14.602638 24378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } }
I20260924 10:44:14.603559 24378 leader_election.cc:288] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [CANDIDATE]: Term 1 election: Requested vote from peers 90a681830cd049c7950912b3fa7caaf6 (127.2.8.61:42729), 6bc61198e44645f682ad42f1dea2a774 (127.2.8.60:45837)
I20260924 10:44:14.604254 24528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6bc61198e44645f682ad42f1dea2a774"
W20260924 10:44:14.604461 24357 tablet.cc:2369] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:44:14.604419 24453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "90a681830cd049c7950912b3fa7caaf6"
I20260924 10:44:14.604619 24453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:14.604490 24528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:14.601963 24411 master_runner.cc:186] Error getting master registration for 127.2.8.62:43191: OK, instance_id { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" instance_seqno: 1790246654108053 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:44:14.605537 24411 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:44:14.606038 24453 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b6f435e26fa4578a69b1a16b2bcf3b3 in term 1.
I20260924 10:44:14.606503 24352 leader_election.cc:302] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b6f435e26fa4578a69b1a16b2bcf3b3, 90a681830cd049c7950912b3fa7caaf6; no voters:
I20260924 10:44:14.606765 24588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:14.606798 24528 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b6f435e26fa4578a69b1a16b2bcf3b3 in term 1.
I20260924 10:44:14.606987 24588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [term 1 LEADER]: Becoming Leader. State: Replica: 1b6f435e26fa4578a69b1a16b2bcf3b3, State: Running, Role: LEADER
I20260924 10:44:14.607327 24588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } }
I20260924 10:44:14.607374 24575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } } }
I20260924 10:44:14.607494 24575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:14.609503 24588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b6f435e26fa4578a69b1a16b2bcf3b3. Latest consensus state: current_term: 1 leader_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } } }
I20260924 10:44:14.609603 24588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:14.610083 24607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260924 10:44:14.610750 24587 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:44:14.610823 24587 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:44:14.614962 24607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:14.625759 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.609057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45916 (local address 127.2.8.62:43191)
0924 10:44:14.609162 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.609166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.612677 (+ 3511us) server_negotiation.cc:407] Connection header received
0924 10:44:14.612812 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.612816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.612868 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.612942 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.615104 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.615791 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.619044 (+ 3253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.619372 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.619444 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.621340 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:14.621344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:14.621347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:14.621374 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:14.623188 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.624225 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.624230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.624234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.624328 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.624658 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.624661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.624663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.624911 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:14.625047 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:14.625420 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 10:44:14.625560 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:44:14.628320 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.607520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41428 (local address 127.2.8.60:45837)
0924 10:44:14.607643 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.607647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.609312 (+ 1665us) server_negotiation.cc:407] Connection header received
0924 10:44:14.611250 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.611255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.611332 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.611397 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.613160 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.613998 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.616550 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.616908 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.617012 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.623218 (+ 6206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:14.623225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:14.623229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:14.623262 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:14.626467 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.626943 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.626946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.626948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.627038 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.627313 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.627320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.627325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.627555 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:14.627696 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:14.628048 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 10:44:14.628169 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 10:44:14.629192 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:14.608619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.2.8.61:42729)
0924 10:44:14.608719 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.608723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.613053 (+ 4330us) server_negotiation.cc:407] Connection header received
0924 10:44:14.615540 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.615543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.615590 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.615638 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.617799 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.618492 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.621006 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:14.621400 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:14.621483 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:14.623619 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:14.623625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:14.623630 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:14.623658 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:14.626218 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.627674 (+ 1456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.627678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.627680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.627764 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:14.628056 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:14.628059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:14.628060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:14.628258 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:14.628388 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:14.628924 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 10:44:14.629063 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:44:14.634243 24453 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6 [term 1 FOLLOWER]: Refusing update from remote peer 1b6f435e26fa4578a69b1a16b2bcf3b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:14.634244 24528 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774 [term 1 FOLLOWER]: Refusing update from remote peer 1b6f435e26fa4578a69b1a16b2bcf3b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:14.634958 24599 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:14.635327 24588 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:14.638690 24575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b6f435e26fa4578a69b1a16b2bcf3b3. Latest consensus state: current_term: 1 leader_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } } }
I20260924 10:44:14.638823 24575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:14.638952 24562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b6f435e26fa4578a69b1a16b2bcf3b3. Latest consensus state: current_term: 1 leader_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } } }
I20260924 10:44:14.639065 24562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:14.644212 24588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } } }
I20260924 10:44:14.644356 24588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:14.646188 24612 mvcc.cc:205] Tried to move back new op lower bound from 7332850297354956800 to 7332850297275105280. Current Snapshot: MvccSnapshot[applied={T|T < 7332850297354956800}]
I20260924 10:44:14.646726 24575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } } }
I20260924 10:44:14.646843 24575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:14.647244 24599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } } }
I20260924 10:44:14.647312 24615 mvcc.cc:205] Tried to move back new op lower bound from 7332850297354956800 to 7332850297275105280. Current Snapshot: MvccSnapshot[applied={T|T < 7332850297354956800}]
I20260924 10:44:14.647375 24599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:14.648276 24562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6f435e26fa4578a69b1a16b2bcf3b3" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 43191 } } peers { permanent_uuid: "90a681830cd049c7950912b3fa7caaf6" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 42729 } } peers { permanent_uuid: "6bc61198e44645f682ad42f1dea2a774" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45837 } } }
I20260924 10:44:14.648387 24562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:14.648643 24614 mvcc.cc:205] Tried to move back new op lower bound from 7332850297354956800 to 7332850297275105280. Current Snapshot: MvccSnapshot[applied={T|T < 7332850297354956800}]
I20260924 10:44:14.651672 24607 catalog_manager.cc:1347] Generated new cluster ID: 9c304a19675948b883093360b5f90f69
I20260924 10:44:14.651733 24607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:14.690305 24607 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:14.691661 24607 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:14.717141 24607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b6f435e26fa4578a69b1a16b2bcf3b3: Generated new TSK 0
I20260924 10:44:14.718182 24607 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:44:14.749536 24560 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:14.746875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.2.8.60:45837)
0924 10:44:14.747010 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.747014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.748433 (+ 1419us) server_negotiation.cc:407] Connection header received
0924 10:44:14.748633 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.748637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.748684 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.748735 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.749390 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41430: BlockingRecv error: recv got EOF from 127.0.0.1:41430 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 10:44:14.750991 24408 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:14.746447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.2.8.62:43191)
0924 10:44:14.746575 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.746580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.750102 (+ 3522us) server_negotiation.cc:407] Connection header received
0924 10:44:14.750279 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.750282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.750333 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.750387 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.750874 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45920: BlockingRecv error: recv got EOF from 127.0.0.1:45920 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 10:44:14.752404 24484 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:14.747791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.2.8.61:42729)
0924 10:44:14.747908 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.747913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.751535 (+ 3622us) server_negotiation.cc:407] Connection header received
0924 10:44:14.751705 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.751708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.751758 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.751811 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.752288 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51704: BlockingRecv error: recv got EOF from 127.0.0.1:51704 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 10:44:14.757171 24484 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:14.755277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.2.8.61:42729)
0924 10:44:14.755406 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.755410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.756169 (+ 759us) server_negotiation.cc:407] Connection header received
0924 10:44:14.756382 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.756386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.756473 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.756528 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.757065 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51706: BlockingRecv error: recv got EOF from 127.0.0.1:51706 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 10:44:14.762356 24484 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:14.760773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.2.8.61:42729)
0924 10:44:14.760873 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.760877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.761466 (+ 589us) server_negotiation.cc:407] Connection header received
0924 10:44:14.761637 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.761641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.761687 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.761735 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.762248 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51708: BlockingRecv error: recv got EOF from 127.0.0.1:51708 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 10:44:14.768204 24484 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:14.766634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.2.8.61:42729)
0924 10:44:14.766734 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.766737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.767256 (+ 519us) server_negotiation.cc:407] Connection header received
0924 10:44:14.767495 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.767498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.767548 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.767601 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.768089 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51714: BlockingRecv error: recv got EOF from 127.0.0.1:51714 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 10:44:14.776062 24484 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:14.774521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.2.8.61:42729)
0924 10:44:14.774662 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.774666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.775139 (+ 473us) server_negotiation.cc:407] Connection header received
0924 10:44:14.775335 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.775339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.775391 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.775457 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.775963 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51718: BlockingRecv error: recv got EOF from 127.0.0.1:51718 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 10:44:14.788542 24484 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:14.786751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51734 (local address 127.2.8.61:42729)
0924 10:44:14.786875 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.786879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.787457 (+ 578us) server_negotiation.cc:407] Connection header received
0924 10:44:14.787690 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.787694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.787750 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.787808 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.788420 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51734: BlockingRecv error: recv got EOF from 127.0.0.1:51734 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 10:44:14.809329 24484 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:14.807316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51740 (local address 127.2.8.61:42729)
0924 10:44:14.807637 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.807642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.808226 (+ 584us) server_negotiation.cc:407] Connection header received
0924 10:44:14.808426 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.808430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.808485 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.808546 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.809202 (+ 656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51740: BlockingRecv error: recv got EOF from 127.0.0.1:51740 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 10:44:14.845208 24484 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:14.844136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.2.8.61:42729)
0924 10:44:14.844319 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.844322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.844430 (+ 108us) server_negotiation.cc:407] Connection header received
0924 10:44:14.844503 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.844505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.844553 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.844625 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.845111 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51748: BlockingRecv error: recv got EOF from 127.0.0.1:51748 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260924 10:44:14.912701 24484 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:14.911372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.2.8.61:42729)
0924 10:44:14.911539 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:14.911544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:14.911717 (+ 173us) server_negotiation.cc:407] Connection header received
0924 10:44:14.911904 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:14.911907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:14.911966 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:14.912066 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:14.912556 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51758: BlockingRecv error: recv got EOF from 127.0.0.1:51758 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 10:44:15.044279 24484 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:15.043079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.2.8.61:42729)
0924 10:44:15.043236 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:15.043241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:15.043355 (+ 114us) server_negotiation.cc:407] Connection header received
0924 10:44:15.043555 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:15.043565 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:15.043614 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:15.043708 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:15.044159 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51766: BlockingRecv error: recv got EOF from 127.0.0.1:51766 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 10:44:15.303609 24484 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:15.302389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.2.8.61:42729)
0924 10:44:15.302550 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:15.302556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:15.302642 (+ 86us) server_negotiation.cc:407] Connection header received
0924 10:44:15.302842 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:15.302848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:15.302902 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:15.302996 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:15.303488 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51778: BlockingRecv error: recv got EOF from 127.0.0.1:51778 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 10:44:15.594635 24573 catalog_manager.cc:1259] Loaded cluster ID: 9c304a19675948b883093360b5f90f69
I20260924 10:44:15.594712 24573 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774: loading cluster ID for follower catalog manager: success
I20260924 10:44:15.596406 24573 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774: acquiring CA information for follower catalog manager: success
I20260924 10:44:15.597622 24573 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6bc61198e44645f682ad42f1dea2a774: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:44:15.612221 24587 catalog_manager.cc:1259] Loaded cluster ID: 9c304a19675948b883093360b5f90f69
I20260924 10:44:15.612300 24587 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6: loading cluster ID for follower catalog manager: success
I20260924 10:44:15.614007 24587 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6: acquiring CA information for follower catalog manager: success
I20260924 10:44:15.615199 24587 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 90a681830cd049c7950912b3fa7caaf6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:44:15.821925 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:15.817721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51794 (local address 127.2.8.61:42729)
0924 10:44:15.817960 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:15.817967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:15.818076 (+ 109us) server_negotiation.cc:407] Connection header received
0924 10:44:15.818258 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:15.818262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:15.818393 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:15.818518 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:15.818788 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:15.819315 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:15.820221 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:15.820472 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:15.820583 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:15.820915 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:15.821504 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:15.821645 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:15.821706 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 10:44:15.821763 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":62,"threads_started":1}
I20260924 10:44:15.828104 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:15.824773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.2.8.60:45837)
0924 10:44:15.825073 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:15.825077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:15.825090 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:44:15.825163 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:15.825165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:15.825273 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:15.825365 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:15.825550 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:15.826042 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:15.826791 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:15.827002 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:15.827073 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:15.827312 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:15.827698 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:15.827786 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:15.827860 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 10:44:15.827903 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260924 10:44:15.834806 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:15.831415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.2.8.61:42729)
0924 10:44:15.831537 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:15.831542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:15.831556 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:44:15.831646 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:15.831649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:15.831695 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:15.831767 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:15.832037 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:15.832773 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:15.833733 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:15.833960 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:15.834072 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:15.834322 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:15.834385 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:15.834466 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:15.834593 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 10:44:15.834652 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 10:44:15.835558 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:15.831224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41452 (local address 127.2.8.60:45837)
0924 10:44:15.831327 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:15.831331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:15.831345 (+ 14us) server_negotiation.cc:407] Connection header received
0924 10:44:15.831390 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:15.831393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:15.831440 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:15.831807 (+ 367us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:15.831895 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:15.832679 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:15.834110 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:15.834440 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:15.834853 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:15.834915 (+ 62us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:15.834990 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:15.835324 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:15.835363 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 10:44:15.835425 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 10:44:15.835724 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:15.830911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.2.8.62:43191)
0924 10:44:15.831538 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:15.831543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:15.831556 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:44:15.831651 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:15.831653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:15.831808 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:15.831923 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:15.832166 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:15.832693 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:15.834110 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:15.834461 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:15.834560 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:15.834893 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:15.835227 (+ 334us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:15.835443 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:15.835521 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 10:44:15.835578 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":67,"threads_started":1}
I20260924 10:44:15.838991 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 24336
I20260924 10:44:15.845391 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 24411
I20260924 10:44:15.851181 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 24485
2026-09-24T10:44:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3076 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 10:44:15 dist-test-slave-j38h krb5kdc[24702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 10:44:15 dist-test-slave-j38h krb5kdc[24702](info): set up 2 sockets
Sep 24 10:44:15 dist-test-slave-j38h krb5kdc[24702](info): commencing operation
krb5kdc: starting...
W20260924 10:44:16.889438 2080 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/test-user.keytab.
Sep 24 10:44:16 dist-test-slave-j38h krb5kdc[24702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T10:44:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T10:44:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.8.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
Entry for principal kudu/127.2.8.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab.
I20260924 10:44:17.011911 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:32993
--webserver_interface=127.2.8.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:32993,127.2.8.61:38957,127.2.8.60:45923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 10:44:17.120463 24715 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:44:17.120822 24715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:17.120901 24715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:17.121042 24715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:17.125097 24715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:17.125185 24715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:17.125226 24715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:17.125275 24715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:17.125315 24715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:17.130524 24715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:32993,127.2.8.61:38957,127.2.8.60:45923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.62.keytab
--principal=kudu/127.2.8.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.62:32993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.8.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:17.131680 24715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:17.132696 24715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:17.141091 24720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:17.141094 24723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:17.141094 24721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:17.141321 24715 server_base.cc:1034] running on GCE node
I20260924 10:44:17.141990 24715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:17.143093 24715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:17.144282 24715 hybrid_clock.cc:648] HybridClock initialized: now 1790246657144275 us; error 29 us; skew 500 ppm
Sep 24 10:44:17 dist-test-slave-j38h krb5kdc[24702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:17.146965 24715 init.cc:377] Logged in from keytab as kudu/127.2.8.62@KRBTEST.COM (short username kudu)
I20260924 10:44:17.147806 24715 webserver.cc:460] Webserver started at http://127.2.8.62:43733/ using document root <none> and password file <none>
I20260924 10:44:17.148391 24715 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:17.148458 24715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:17.148617 24715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:17.150626 24715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/data/instance:
uuid: "230f19f93a4f431396c58a7fbb5620d8"
format_stamp: "Formatted at 2026-09-24 10:44:17 on dist-test-slave-j38h"
server_key: "8346bd5c773ae12f84ab28866012d893"
server_key_iv: "1c9d5edfd581256537f973d74fb70bf4"
server_key_version: "encryptionkey@0"
I20260924 10:44:17.151122 24715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/wal/instance:
uuid: "230f19f93a4f431396c58a7fbb5620d8"
format_stamp: "Formatted at 2026-09-24 10:44:17 on dist-test-slave-j38h"
server_key: "8346bd5c773ae12f84ab28866012d893"
server_key_iv: "1c9d5edfd581256537f973d74fb70bf4"
server_key_version: "encryptionkey@0"
I20260924 10:44:17.154752 24715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 10:44:17.157078 24730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:17.158380 24715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:44:17.158485 24715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/wal
uuid: "230f19f93a4f431396c58a7fbb5620d8"
format_stamp: "Formatted at 2026-09-24 10:44:17 on dist-test-slave-j38h"
server_key: "8346bd5c773ae12f84ab28866012d893"
server_key_iv: "1c9d5edfd581256537f973d74fb70bf4"
server_key_version: "encryptionkey@0"
I20260924 10:44:17.158571 24715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:17.174122 24715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:17.176883 24715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:17.177111 24715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:17.183946 24715 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.62:32993
I20260924 10:44:17.184226 24782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.62:32993 every 8 connection(s)
I20260924 10:44:17.184937 24715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/data/info.pb
I20260924 10:44:17.187597 24783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:17.187986 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 24715
I20260924 10:44:17.188128 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-0/wal/instance
I20260924 10:44:17.188462 2080 external_mini_cluster.cc:1428] Setting key a96c97765d10cb05ae8102ac4a38f2b9
I20260924 10:44:17.192322 24783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.8.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.61@KRBTEST.COM" created.
Sep 24 10:44:17 dist-test-slave-j38h krb5kdc[24702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:44:17.208726 24787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.195183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.2.8.62:32993)
0924 10:44:17.195759 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.195776 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.195930 (+ 154us) server_negotiation.cc:407] Connection header received
0924 10:44:17.197316 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.197321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.197533 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.197677 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.198478 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.199458 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.200547 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.201227 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.201501 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.203962 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:17.203968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:17.203977 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:17.203996 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:17.205656 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.206139 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.206145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.206146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.206251 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.206570 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.206573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.206575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.206927 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:17.207068 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:17.207415 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 10:44:17.207687 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":460,"spinlock_wait_cycles":39168,"thread_start_us":189,"threads_started":1}
I20260924 10:44:17.208741 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.195184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:32993 (local address 127.0.0.1:44480)
0924 10:44:17.195754 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.195806 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:17.196956 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:17.197712 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:17.197716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:17.198072 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:17.198320 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.198333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.199576 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.199579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.200421 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.200431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.201324 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.201388 (+ 64us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.201603 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.202193 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:17.202217 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:17.203805 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:17.205782 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.205792 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.205802 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.206030 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.206346 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.206350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.206352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.206466 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.207041 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:17.207047 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:17.207188 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 10:44:17.207357 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 10:44:17.207736 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":127,"thread_start_us":141,"threads_started":1}
I20260924 10:44:17.211033 24783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:44:17.211686 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.211455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:38957 (local address 127.0.0.1:49898)
0924 10:44:17.211564 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.211629 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 10:44:17.211861 24732 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.8.61:38957: connect: Connection refused (error 111)
I20260924 10:44:17.212574 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.212371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:38957 (local address 127.0.0.1:49906)
0924 10:44:17.212480 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.212527 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 10:44:17.212703 24783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.61:38957: Network error: Client connection negotiation failed: client connection to 127.2.8.61:38957: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.61.keytab.
Entry for principal kudu/127.2.8.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.61.keytab.
I20260924 10:44:17.216372 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.61:38957
--webserver_interface=127.2.8.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.61.keytab
--principal=kudu/127.2.8.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:32993,127.2.8.61:38957,127.2.8.60:45923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 10:44:17.261950 24783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } attempt: 1
I20260924 10:44:17.262843 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.262548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:38957 (local address 127.0.0.1:49920)
0924 10:44:17.262692 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.262761 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 10:44:17.263582 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.263368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:38957 (local address 127.0.0.1:49928)
0924 10:44:17.263470 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.263518 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 10:44:17.263756 24783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.61:38957: Network error: Client connection negotiation failed: client connection to 127.2.8.61:38957: connect: Connection refused (error 111)
W20260924 10:44:17.324527 24790 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:44:17.324860 24790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:17.324945 24790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:17.325064 24790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 10:44:17.329139 24790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:17.329223 24790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:17.329262 24790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:17.329298 24790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:17.329334 24790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:17.331934 24783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } attempt: 2
I20260924 10:44:17.332708 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.332404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:38957 (local address 127.0.0.1:49942)
0924 10:44:17.332569 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.332639 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 10:44:17.333436 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.333228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:38957 (local address 127.0.0.1:49952)
0924 10:44:17.333330 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.333377 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.61:38957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:44:17.333563 24783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.61:38957: Network error: Client connection negotiation failed: client connection to 127.2.8.61:38957: connect: Connection refused (error 111)
I20260924 10:44:17.334956 24790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:32993,127.2.8.61:38957,127.2.8.60:45923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.61.keytab
--principal=kudu/127.2.8.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.61:38957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.8.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:17.336051 24790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:17.337143 24790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 10:44:17.345356 24795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:17.345438 24796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:17.345353 24798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:17.345534 24790 server_base.cc:1034] running on GCE node
I20260924 10:44:17.346172 24790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:17.347273 24790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:17.348469 24790 hybrid_clock.cc:648] HybridClock initialized: now 1790246657348433 us; error 48 us; skew 500 ppm
Sep 24 10:44:17 dist-test-slave-j38h krb5kdc[24702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:17.351130 24790 init.cc:377] Logged in from keytab as kudu/127.2.8.61@KRBTEST.COM (short username kudu)
I20260924 10:44:17.352041 24790 webserver.cc:460] Webserver started at http://127.2.8.61:36335/ using document root <none> and password file <none>
I20260924 10:44:17.352612 24790 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:17.352700 24790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:17.352929 24790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:17.354949 24790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/data/instance:
uuid: "dca4dc1d229140dc91c2146441fabb01"
format_stamp: "Formatted at 2026-09-24 10:44:17 on dist-test-slave-j38h"
server_key: "95187194a3a7c077d203081f244a9351"
server_key_iv: "dc5157f5aa222b106f9ebf937631a1ea"
server_key_version: "encryptionkey@0"
I20260924 10:44:17.355527 24790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/wal/instance:
uuid: "dca4dc1d229140dc91c2146441fabb01"
format_stamp: "Formatted at 2026-09-24 10:44:17 on dist-test-slave-j38h"
server_key: "95187194a3a7c077d203081f244a9351"
server_key_iv: "dc5157f5aa222b106f9ebf937631a1ea"
server_key_version: "encryptionkey@0"
I20260924 10:44:17.359148 24790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 10:44:17.361532 24805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:17.362805 24790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 10:44:17.362934 24790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/wal
uuid: "dca4dc1d229140dc91c2146441fabb01"
format_stamp: "Formatted at 2026-09-24 10:44:17 on dist-test-slave-j38h"
server_key: "95187194a3a7c077d203081f244a9351"
server_key_iv: "dc5157f5aa222b106f9ebf937631a1ea"
server_key_version: "encryptionkey@0"
I20260924 10:44:17.363044 24790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:17.389613 24790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:17.392441 24790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:17.392647 24790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:17.399675 24790 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.61:38957
I20260924 10:44:17.399962 24857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.61:38957 every 8 connection(s)
I20260924 10:44:17.400643 24790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/data/info.pb
I20260924 10:44:17.402657 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 24790
I20260924 10:44:17.402797 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-1/wal/instance
I20260924 10:44:17.403081 2080 external_mini_cluster.cc:1428] Setting key bf325bbe898dea5df82922350e60b97b
I20260924 10:44:17.403580 24858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:17.407280 24858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.8.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.8.60@KRBTEST.COM" created.
Sep 24 10:44:17 dist-test-slave-j38h krb5kdc[24702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.61@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:44:17.421490 24787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.409169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44490 (local address 127.2.8.62:32993)
0924 10:44:17.409334 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.409341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.410190 (+ 849us) server_negotiation.cc:407] Connection header received
0924 10:44:17.411101 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.411104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.411153 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.411231 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.412469 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.413057 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.414844 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.415276 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.415397 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.417861 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:17.417867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:17.417875 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:17.417902 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:17.419391 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.419896 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.419901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.419903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.420001 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.420342 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.420346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.420348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.420535 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:17.420644 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:17.421184 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 10:44:17.421316 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 10:44:17.422369 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.409550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:32993 (local address 127.0.0.1:44490)
0924 10:44:17.410017 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.410067 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:17.410905 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:17.411386 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:17.411393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:17.411866 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:17.412328 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.412341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.413221 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.413224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.414679 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.414696 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.415381 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.415435 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.415579 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.416200 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:17.416216 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:17.417688 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:17.419513 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.419519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.419533 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.419792 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.420105 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.420108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.420110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.420247 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.420642 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:17.420649 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:17.420895 (+ 246us) client_negotiation.cc:769] Sending connection context
0924 10:44:17.421124 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 10:44:17.421452 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":137,"threads_started":1}
I20260924 10:44:17.423722 24858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:44:17.424714 24783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.8.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.60.keytab.
Entry for principal kudu/127.2.8.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.60.keytab.
I20260924 10:44:17.429235 2080 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
/tmp/dist-test-task2SD_Oz/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.60:45923
--webserver_interface=127.2.8.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.60.keytab
--principal=kudu/127.2.8.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.8.20:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.8.62:32993,127.2.8.61:38957,127.2.8.60:45923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 10:44:17 dist-test-slave-j38h krb5kdc[24702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.61@KRBTEST.COM for kudu/127.2.8.61@KRBTEST.COM
Sep 24 10:44:17 dist-test-slave-j38h krb5kdc[24702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for kudu/127.2.8.61@KRBTEST.COM
I20260924 10:44:17.433808 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.424207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:38957 (local address 127.0.0.1:49964)
0924 10:44:17.424325 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.424340 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:17.424411 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:17.425127 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:17.425131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:17.425331 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:17.425479 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.425487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.426624 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.426628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.427280 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.427293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.427841 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.427855 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.427970 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.428467 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:17.428481 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:17.430064 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:17.431896 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.431902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.431906 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.432182 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.432556 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.432560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.432562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.432665 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.433245 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:17.433248 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:17.433361 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 10:44:17.433486 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 10:44:17.433609 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 10:44:17.434037 24863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.424330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.2.8.61:38957)
0924 10:44:17.424634 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.424643 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.424659 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:44:17.424715 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.424720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.424908 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.425144 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.425572 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.426518 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.427375 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.427731 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.427846 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.430230 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:17.430242 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:17.430261 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:17.430291 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:17.431808 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.432314 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.432323 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.432328 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.432437 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.432774 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.432780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.432785 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.433138 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:17.433280 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:17.433598 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 10:44:17.433801 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":123,"threads_started":1}
I20260924 10:44:17.435616 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.425104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:38957 (local address 127.0.0.1:49974)
0924 10:44:17.425271 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.425285 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:17.425357 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:17.425768 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:17.425771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:17.425937 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:17.426031 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.426037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.426890 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.426893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.427903 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.427912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.428510 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.428530 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.428645 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.429174 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:17.429189 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:17.430958 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:17.434082 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.434086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.434090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.434357 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.434654 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.434658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.434660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.434746 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.435153 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:17.435157 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:17.435254 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 10:44:17.435334 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 10:44:17.435473 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 10:44:17.436182 24864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.425127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49974 (local address 127.2.8.61:38957)
0924 10:44:17.425564 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.425574 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.425586 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:44:17.425613 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.425615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.425654 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.425715 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.426142 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.426774 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.428030 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.428342 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.428490 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.431357 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:17.431363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:17.431365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:17.431393 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:17.433931 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.434480 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.434484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.434485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.434564 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.434843 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.434847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.434848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.435037 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:17.435881 (+ 844us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:17.435952 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 10:44:17.436052 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":99,"threads_started":1}
I20260924 10:44:17.436306 24858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:44:17.436951 24783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:44:17.437054 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.436776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45923 (local address 127.0.0.1:44018)
0924 10:44:17.436883 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.436954 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 10:44:17.437305 24807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
I20260924 10:44:17.437745 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.437438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45923 (local address 127.0.0.1:44022)
0924 10:44:17.437631 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.437680 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260924 10:44:17.438279 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.438019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45923 (local address 127.0.0.1:44028)
0924 10:44:17.438144 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.438206 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 10:44:17.438411 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.438165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45923 (local address 127.0.0.1:44040)
0924 10:44:17.438301 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.438352 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 10:44:17.438556 24783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:45923: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
W20260924 10:44:17.438674 24858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:45923: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
I20260924 10:44:17.469697 24783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } attempt: 1
I20260924 10:44:17.470557 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.470210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45923 (local address 127.0.0.1:44048)
0924 10:44:17.470379 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.470466 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 10:44:17.471282 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.471037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45923 (local address 127.0.0.1:44062)
0924 10:44:17.471164 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.471220 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 10:44:17.471441 24783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:45923: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
I20260924 10:44:17.487960 24858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } attempt: 1
I20260924 10:44:17.488835 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.488513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45923 (local address 127.0.0.1:44070)
0924 10:44:17.488674 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.488746 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 10:44:17.489588 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.489359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45923 (local address 127.0.0.1:44080)
0924 10:44:17.489476 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.489525 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 10:44:17.489725 24858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:45923: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
W20260924 10:44:17.543782 24865 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 10:44:17.544049 24865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 10:44:17.544113 24865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 10:44:17.544200 24865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 10:44:17.546619 24783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } attempt: 2
I20260924 10:44:17.547464 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.547168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45923 (local address 127.0.0.1:44082)
0924 10:44:17.547319 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.547382 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 10:44:17.548173 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.547964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45923 (local address 127.0.0.1:44086)
0924 10:44:17.548070 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.548120 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 10:44:17.548296 24783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:45923: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
W20260924 10:44:17.548647 24865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 10:44:17.548720 24865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 10:44:17.548782 24865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 10:44:17.548833 24865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 10:44:17.548870 24865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 10:44:17.554152 24865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.8.20:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.8.62:32993,127.2.8.61:38957,127.2.8.60:45923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/krb5kdc/kudu_127.2.8.60.keytab
--principal=kudu/127.2.8.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.8.60:45923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.8.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 09:59:18 UTC on ccycloud.quasar-bneeji.root.comops.site
build id 4669761
I20260924 10:44:17.555174 24865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 10:44:17.556136 24865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 10:44:17.557898 24858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } attempt: 2
I20260924 10:44:17.558655 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.558309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45923 (local address 127.0.0.1:44102)
0924 10:44:17.558490 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.558564 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 10:44:17.559361 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.559143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45923 (local address 127.0.0.1:44112)
0924 10:44:17.559249 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.559300 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 10:44:17.559489 24858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.8.60:45923: Network error: Client connection negotiation failed: client connection to 127.2.8.60:45923: connect: Connection refused (error 111)
W20260924 10:44:17.564594 24871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 10:44:17.564635 24874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:17.564796 24865 server_base.cc:1034] running on GCE node
W20260924 10:44:17.564801 24872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 10:44:17.565493 24865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 10:44:17.566591 24865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 10:44:17.567770 24865 hybrid_clock.cc:648] HybridClock initialized: now 1790246657567755 us; error 26 us; skew 500 ppm
Sep 24 10:44:17 dist-test-slave-j38h krb5kdc[24702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 10:44:17.570425 24865 init.cc:377] Logged in from keytab as kudu/127.2.8.60@KRBTEST.COM (short username kudu)
I20260924 10:44:17.571273 24865 webserver.cc:460] Webserver started at http://127.2.8.60:38109/ using document root <none> and password file <none>
I20260924 10:44:17.571802 24865 fs_manager.cc:359] Metadata directory not provided
I20260924 10:44:17.571889 24865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 10:44:17.572105 24865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 10:44:17.574095 24865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/data/instance:
uuid: "66800f30d6a641e18ffb243fdf3d5011"
format_stamp: "Formatted at 2026-09-24 10:44:17 on dist-test-slave-j38h"
server_key: "1b23e8e98e17e69ce56093293f6fdf09"
server_key_iv: "d35fda42437a5605168d85f2fef476c9"
server_key_version: "encryptionkey@0"
I20260924 10:44:17.574671 24865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/wal/instance:
uuid: "66800f30d6a641e18ffb243fdf3d5011"
format_stamp: "Formatted at 2026-09-24 10:44:17 on dist-test-slave-j38h"
server_key: "1b23e8e98e17e69ce56093293f6fdf09"
server_key_iv: "d35fda42437a5605168d85f2fef476c9"
server_key_version: "encryptionkey@0"
I20260924 10:44:17.578833 24865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 10:44:17.581137 24881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 10:44:17.582280 24865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 10:44:17.582407 24865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/data,/tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/wal
uuid: "66800f30d6a641e18ffb243fdf3d5011"
format_stamp: "Formatted at 2026-09-24 10:44:17 on dist-test-slave-j38h"
server_key: "1b23e8e98e17e69ce56093293f6fdf09"
server_key_iv: "d35fda42437a5605168d85f2fef476c9"
server_key_version: "encryptionkey@0"
I20260924 10:44:17.582526 24865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 10:44:17.611922 24865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 10:44:17.614938 24865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 10:44:17.615149 24865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 10:44:17.621845 24865 rpc_server.cc:307] RPC server started. Bound to: 127.2.8.60:45923
I20260924 10:44:17.622130 24933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.8.60:45923 every 8 connection(s)
I20260924 10:44:17.622954 24865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/data/info.pb
I20260924 10:44:17.625792 2080 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu as pid 24865
I20260924 10:44:17.625932 2080 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2SD_Oz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790246374820021-2080-0/minicluster-data/master-2/wal/instance
I20260924 10:44:17.626040 24934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 10:44:17.626288 2080 external_mini_cluster.cc:1428] Setting key 3109c2c3a43dccb6cf4ab9031545f523
I20260924 10:44:17.631111 24934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 10:44:17 dist-test-slave-j38h krb5kdc[24702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.60@KRBTEST.COM
Sep 24 10:44:17 dist-test-slave-j38h krb5kdc[24702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.60@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:44:17.641189 24937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.627588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44128 (local address 127.2.8.60:45923)
0924 10:44:17.627987 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.628005 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.628047 (+ 42us) server_negotiation.cc:407] Connection header received
0924 10:44:17.628747 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.628770 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.629230 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.629601 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.630225 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.631691 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.632531 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.633173 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.633293 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.636044 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:17.636051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:17.636062 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:17.636092 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:17.637971 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.638482 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.638495 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.638499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.638640 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.638912 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.638916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.638924 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.639450 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:17.639656 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:17.639891 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 10:44:17.640199 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
I20260924 10:44:17.641940 24787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.632944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44506 (local address 127.2.8.62:32993)
0924 10:44:17.633129 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.633134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.633710 (+ 576us) server_negotiation.cc:407] Connection header received
0924 10:44:17.633851 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.633855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.633906 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.634010 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.634635 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.635252 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.636238 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.636469 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.636578 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.638628 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:17.638632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:17.638634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:17.638659 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:17.640169 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.640582 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.640585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.640587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.640674 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.640926 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.640929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.640931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.641170 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:17.641277 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:17.641676 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 10:44:17.641786 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 10:44:17.641968 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.632872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.62:32993 (local address 127.0.0.1:44506)
0924 10:44:17.633561 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.633628 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:17.633744 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:17.634070 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:17.634075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:17.634396 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:17.634515 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.634521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.635377 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.635380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.636115 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.636126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.636568 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.636583 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.636720 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.637172 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:17.637183 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:17.638498 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:17.640272 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.640279 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.640283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.640487 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.640772 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.640774 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.640776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.640842 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.641287 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:17.641296 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:17.641440 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 10:44:17.641605 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 10:44:17.641779 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":174,"threads_started":1}
I20260924 10:44:17.643015 24934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:44:17.647452 24783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } attempt: 3
Sep 24 10:44:17 dist-test-slave-j38h krb5kdc[24702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.60@KRBTEST.COM for kudu/127.2.8.61@KRBTEST.COM
Sep 24 10:44:17 dist-test-slave-j38h krb5kdc[24702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.62@KRBTEST.COM
I20260924 10:44:17.650689 24858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } attempt: 3
I20260924 10:44:17.652202 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.643491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.61:38957 (local address 127.0.0.1:49984)
0924 10:44:17.643605 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.643619 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:17.643687 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:17.644089 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:17.644091 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:17.644251 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:17.644333 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.644338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.645101 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.645105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.645619 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.645627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.646099 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.646113 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.646200 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.646767 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:17.646780 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:17.648476 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:17.650469 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.650472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.650474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.650718 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.651103 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.651106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.651108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.651208 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.651690 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:17.651696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:17.651788 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 10:44:17.651905 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 10:44:17.652028 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 10:44:17.652211 24864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.643736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49984 (local address 127.2.8.61:38957)
0924 10:44:17.643871 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.643875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.643887 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:44:17.643931 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.643935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.643982 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.644076 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.644443 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.644962 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.645742 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.646001 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.646111 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.648600 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:17.648603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:17.648606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:17.648634 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:17.650341 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.650884 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.650890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.650894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.650986 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.651318 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.651322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.651323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.651547 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:17.651615 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:17.651965 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 10:44:17.652074 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 10:44:17.652609 24787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.643474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44516 (local address 127.2.8.62:32993)
0924 10:44:17.643572 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.643576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.643620 (+ 44us) server_negotiation.cc:407] Connection header received
0924 10:44:17.643753 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.643756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.643801 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.643869 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.644480 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.644965 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.645746 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.645979 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.646090 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.648779 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:17.648785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:17.648790 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:17.648816 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:17.650323 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.650841 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.650845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.650847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.650951 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.651237 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.651240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.651242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.651427 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:17.651651 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:17.652331 (+ 680us) server_negotiation.cc:299] Negotiation successful
0924 10:44:17.652445 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 10:44:17 dist-test-slave-j38h krb5kdc[24702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.62@KRBTEST.COM for kudu/127.2.8.60@KRBTEST.COM
I20260924 10:44:17.654743 24934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } has no permanent_uuid. Determining permanent_uuid...
I20260924 10:44:17.657389 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.647862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45923 (local address 127.0.0.1:44142)
0924 10:44:17.648085 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.648097 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:17.648183 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:17.648448 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:17.648454 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:17.648629 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:17.648744 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.648752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.650633 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.650638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.651431 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.651438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.651979 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.651995 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.652091 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.652507 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:17.652520 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:17.653909 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:17.655836 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.655839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.655841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.656076 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.656334 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.656338 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.656341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.656432 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.656856 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:17.656862 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:17.656996 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 10:44:17.657086 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 10:44:17.657222 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260924 10:44:17.657984 24937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.647967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.2.8.60:45923)
0924 10:44:17.648060 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.648065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.648163 (+ 98us) server_negotiation.cc:407] Connection header received
0924 10:44:17.648281 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.648284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.648328 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.648412 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.649321 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.650525 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.651548 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.651865 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.652294 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.654059 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:17.654063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:17.654065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:17.654086 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:17.655719 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.656173 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.656175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.656176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.656239 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.656537 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.656540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.656542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.656753 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:17.657655 (+ 902us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:17.657726 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 10:44:17.657831 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Sep 24 10:44:17 dist-test-slave-j38h krb5kdc[24702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.61@KRBTEST.COM for kudu/127.2.8.60@KRBTEST.COM
Sep 24 10:44:17 dist-test-slave-j38h krb5kdc[24702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.8.61@KRBTEST.COM
Sep 24 10:44:17 dist-test-slave-j38h krb5kdc[24702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790246657, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.8.60@KRBTEST.COM for kudu/127.2.8.60@KRBTEST.COM
I20260924 10:44:17.661815 24783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8: Bootstrap starting.
I20260924 10:44:17.664943 24783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:17.665124 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.651076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45923 (local address 127.0.0.1:44148)
0924 10:44:17.651344 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.651359 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:17.651421 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:17.653316 (+ 1895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:17.653319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:17.653492 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:17.653589 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.653600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.655327 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.655331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.656066 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.656080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.657116 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.657131 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.657215 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.657631 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:17.657650 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:17.661312 (+ 3662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:17.663492 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.663495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.663497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.663735 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.664036 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.664039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.664041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.664160 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.664579 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:17.664582 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:17.664678 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 10:44:17.664763 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 10:44:17.664910 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":18}
I20260924 10:44:17.665598 24864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.653879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49988 (local address 127.2.8.61:38957)
0924 10:44:17.654152 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.654157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.654170 (+ 13us) server_negotiation.cc:407] Connection header received
0924 10:44:17.654208 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.654211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.654257 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.654615 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.655417 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.656373 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.657936 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.658654 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.658731 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.661649 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:17.661654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:17.661660 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:17.661687 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:17.663492 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.664013 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.664018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.664021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.664122 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.664451 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.664455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.664459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.664693 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:17.664895 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:17.665316 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 10:44:17.665431 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":210,"spinlock_wait_cycles":1280}
I20260924 10:44:17.665930 24783 log.cc:824] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:17.666224 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.655468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.8.60:45923 (local address 127.0.0.1:44162)
0924 10:44:17.656034 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0924 10:44:17.656048 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 10:44:17.656120 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 10:44:17.656755 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 10:44:17.656758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 10:44:17.656919 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 10:44:17.657028 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.657035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.658085 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.658090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.658867 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 10:44:17.658880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.659406 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.659435 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 10:44:17.659510 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.659976 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 10:44:17.660003 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 10:44:17.662235 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 10:44:17.664797 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.664799 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.664801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.665052 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.665318 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 10:44:17.665322 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 10:44:17.665325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 10:44:17.665399 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 10:44:17.665782 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 10:44:17.665785 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 10:44:17.665863 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 10:44:17.665944 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 10:44:17.666070 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492}
I20260924 10:44:17.666735 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.656246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44162 (local address 127.2.8.60:45923)
0924 10:44:17.656556 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.656565 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.656577 (+ 12us) server_negotiation.cc:407] Connection header received
0924 10:44:17.656607 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.656609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.656645 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.657123 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.657231 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.657974 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.658997 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.659301 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.659701 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.662821 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:17.662825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:17.662828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:17.662859 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:17.664702 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.665151 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.665153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.665155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.665232 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.665501 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.665504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.665506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.665685 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:17.666387 (+ 702us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:17.666464 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 10:44:17.666580 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
I20260924 10:44:17.667024 24939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.651543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44148 (local address 127.2.8.60:45923)
0924 10:44:17.653074 (+ 1531us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.653084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.653100 (+ 16us) server_negotiation.cc:407] Connection header received
0924 10:44:17.653134 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.653137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.653175 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.653263 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.653987 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.655202 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.656460 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.657001 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.657416 (+ 415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.661473 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:17.661476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:17.661479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:17.661503 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:17.663346 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.663851 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.663854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.663856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.663937 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.664258 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.664261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.664263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.664469 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:17.665912 (+ 1443us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:17.666003 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 10:44:17.666867 (+ 864us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1468,"thread_start_us":131,"threads_started":1}
I20260924 10:44:17.669118 2080 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 10:44:17.669342 24783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8: No bootstrap required, opened a new log
I20260924 10:44:17.669860 24934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011: Bootstrap starting.
I20260924 10:44:17.669858 24858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01: Bootstrap starting.
I20260924 10:44:17.670303 24757 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "230f19f93a4f431396c58a7fbb5620d8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44516
I20260924 10:44:17.672158 24858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:17.672343 24934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011: Neither blocks nor log segments found. Creating new log.
I20260924 10:44:17.673012 24858 log.cc:824] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:17.673190 24934 log.cc:824] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011: Log is configured to *not* fsync() on all Append() calls
I20260924 10:44:17.673554 24783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } }
I20260924 10:44:17.673779 24783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230f19f93a4f431396c58a7fbb5620d8, State: Initialized, Role: FOLLOWER
I20260924 10:44:17.674117 24783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } }
I20260924 10:44:17.674600 24944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } } }
I20260924 10:44:17.674757 24944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:17.675108 24858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01: No bootstrap required, opened a new log
I20260924 10:44:17.675518 24783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:17.675561 24757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 10:44:17.675668 24757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:17.677070 24757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } }
I20260924 10:44:17.678078 24757 leader_election.cc:288] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [CANDIDATE]: Term 1 election: Requested vote from peers dca4dc1d229140dc91c2146441fabb01 (127.2.8.61:38957), 66800f30d6a641e18ffb243fdf3d5011 (127.2.8.60:45923)
W20260924 10:44:17.678758 24736 tablet.cc:2369] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 10:44:17.679276 24858 raft_consensus.cc:348] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } }
I20260924 10:44:17.679493 24858 raft_consensus.cc:729] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca4dc1d229140dc91c2146441fabb01, State: Initialized, Role: FOLLOWER
I20260924 10:44:17.679805 24858 consensus_queue.cc:260] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } }
I20260924 10:44:17.680307 24934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011: No bootstrap required, opened a new log
W20260924 10:44:17.680882 24955 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:44:17.681039 24955 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:44:17.681131 24858 sys_catalog.cc:564] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:17.681221 24832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "230f19f93a4f431396c58a7fbb5620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dca4dc1d229140dc91c2146441fabb01"
I20260924 10:44:17.681463 24832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:17.682817 24832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 230f19f93a4f431396c58a7fbb5620d8 in term 1.
I20260924 10:44:17.683300 24732 leader_election.cc:302] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 230f19f93a4f431396c58a7fbb5620d8, dca4dc1d229140dc91c2146441fabb01; no voters:
I20260924 10:44:17.683646 24944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 10:44:17.683874 24944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 230f19f93a4f431396c58a7fbb5620d8, State: Running, Role: LEADER
I20260924 10:44:17.684268 24944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } }
I20260924 10:44:17.684861 24961 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } } }
I20260924 10:44:17.685026 24961 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:17.686061 24944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 230f19f93a4f431396c58a7fbb5620d8. Latest consensus state: current_term: 1 leader_uuid: "230f19f93a4f431396c58a7fbb5620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } } }
I20260924 10:44:17.686190 24944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [sys.catalog]: This master's current role is: LEADER
W20260924 10:44:17.686853 24972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:44:17.686962 24972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:44:17.686872 24934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } }
I20260924 10:44:17.687165 24934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66800f30d6a641e18ffb243fdf3d5011, State: Initialized, Role: FOLLOWER
I20260924 10:44:17.687502 24934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } }
I20260924 10:44:17.688022 24978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 10:44:17.688835 24934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 10:44:17.688884 24908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "230f19f93a4f431396c58a7fbb5620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "66800f30d6a641e18ffb243fdf3d5011"
I20260924 10:44:17.689111 24908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011 [term 0 FOLLOWER]: Advancing to term 1
I20260924 10:44:17.690616 24908 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 230f19f93a4f431396c58a7fbb5620d8 in term 1.
I20260924 10:44:17.690882 24980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } } }
I20260924 10:44:17.691357 24978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 10:44:17.691484 24980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:17.691249 24715 master_runner.cc:186] Error getting master registration for 127.2.8.60:45923: OK, instance_id { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" instance_seqno: 1790246657620014 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:44:17.694562 24715 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 10:44:17.694332 24790 master_runner.cc:186] Error getting master registration for 127.2.8.60:45923: OK, instance_id { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" instance_seqno: 1790246657620014 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 10:44:17.698318 24790 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 10:44:17.702907 24991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 10:44:17.703008 24991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 10:44:17.707533 24787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.686462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44518 (local address 127.2.8.62:32993)
0924 10:44:17.687669 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.687673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.692796 (+ 5123us) server_negotiation.cc:407] Connection header received
0924 10:44:17.693159 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.693163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.693210 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.693284 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.699263 (+ 5979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.700027 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.701474 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.701780 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.701853 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.703765 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:17.703769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:17.703772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:17.703800 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:17.705523 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.706016 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.706020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.706022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.706112 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.706448 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.706453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.706454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.706702 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:17.706834 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:17.707192 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 10:44:17.707327 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1118}
I20260924 10:44:17.710963 24864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.685889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.2.8.61:38957)
0924 10:44:17.686027 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.686031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.693135 (+ 7104us) server_negotiation.cc:407] Connection header received
0924 10:44:17.693301 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.693304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.693345 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.693396 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.694447 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.695211 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.696458 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.696851 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.696933 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.707011 (+ 10078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:17.707015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:17.707018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:17.707046 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:17.709132 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.709572 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.709575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.709578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.709659 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.709964 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.709967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.709970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.710191 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:17.710307 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:17.710673 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 10:44:17.710794 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 10:44:17.711202 24939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:17.686637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44172 (local address 127.2.8.60:45923)
0924 10:44:17.686744 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.686748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.702066 (+ 15318us) server_negotiation.cc:407] Connection header received
0924 10:44:17.702238 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.702241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.702289 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.702344 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.703261 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.704026 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.705388 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:17.705733 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:17.705804 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:17.707436 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 10:44:17.707440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 10:44:17.707443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 10:44:17.707468 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 10:44:17.709362 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.709845 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.709849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.709851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.709952 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 10:44:17.710249 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 10:44:17.710252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 10:44:17.710254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 10:44:17.710466 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 10:44:17.710592 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:17.710931 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 10:44:17.711057 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 10:44:17.722131 24908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011 [term 1 FOLLOWER]: Refusing update from remote peer 230f19f93a4f431396c58a7fbb5620d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:17.722440 24832 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01 [term 1 FOLLOWER]: Refusing update from remote peer 230f19f93a4f431396c58a7fbb5620d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 10:44:17.723121 24944 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:17.723609 24944 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 10:44:17.727818 24961 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 230f19f93a4f431396c58a7fbb5620d8. Latest consensus state: current_term: 1 leader_uuid: "230f19f93a4f431396c58a7fbb5620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } } }
I20260924 10:44:17.727959 24961 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:17.728176 24980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 230f19f93a4f431396c58a7fbb5620d8. Latest consensus state: current_term: 1 leader_uuid: "230f19f93a4f431396c58a7fbb5620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } } }
I20260924 10:44:17.728276 24980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:17.733333 24973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "230f19f93a4f431396c58a7fbb5620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } } }
I20260924 10:44:17.733484 24973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:17.733995 24993 mvcc.cc:205] Tried to move back new op lower bound from 7332850309985153024 to 7332850309878239232. Current Snapshot: MvccSnapshot[applied={T|T < 7332850309985153024}]
I20260924 10:44:17.734655 24961 sys_catalog.cc:455] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "230f19f93a4f431396c58a7fbb5620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } } }
I20260924 10:44:17.734767 24961 sys_catalog.cc:458] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:17.735883 24944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "230f19f93a4f431396c58a7fbb5620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } } }
I20260924 10:44:17.735991 24944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8 [sys.catalog]: This master's current role is: LEADER
I20260924 10:44:17.736896 24980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "230f19f93a4f431396c58a7fbb5620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "230f19f93a4f431396c58a7fbb5620d8" member_type: VOTER last_known_addr { host: "127.2.8.62" port: 32993 } } peers { permanent_uuid: "dca4dc1d229140dc91c2146441fabb01" member_type: VOTER last_known_addr { host: "127.2.8.61" port: 38957 } } peers { permanent_uuid: "66800f30d6a641e18ffb243fdf3d5011" member_type: VOTER last_known_addr { host: "127.2.8.60" port: 45923 } } }
I20260924 10:44:17.737044 24980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 10:44:17.739830 24995 mvcc.cc:205] Tried to move back new op lower bound from 7332850309985153024 to 7332850309878239232. Current Snapshot: MvccSnapshot[applied={T|T < 7332850309985153024}]
I20260924 10:44:17.740017 24994 mvcc.cc:205] Tried to move back new op lower bound from 7332850309985153024 to 7332850309878239232. Current Snapshot: MvccSnapshot[applied={T|T < 7332850309985153024}]
I20260924 10:44:17.741930 24978 catalog_manager.cc:1347] Generated new cluster ID: 297d0c1b956a4ce0b81e246a2bc1ee2d
I20260924 10:44:17.741997 24978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 10:44:17.765367 24978 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 10:44:17.766764 24978 catalog_manager.cc:1504] Loading token signing keys...
I20260924 10:44:17.783553 24978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 230f19f93a4f431396c58a7fbb5620d8: Generated new TSK 0
I20260924 10:44:17.784494 24978 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 10:44:17.820655 24787 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:17.817491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.2.8.62:32993)
0924 10:44:17.817623 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.817628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.819471 (+ 1843us) server_negotiation.cc:407] Connection header received
0924 10:44:17.819672 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.819676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.819728 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.819782 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.820511 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44528: BlockingRecv error: recv got EOF from 127.0.0.1:44528 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 10:44:17.822044 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:17.818318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.2.8.61:38957)
0924 10:44:17.818445 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.818450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.821188 (+ 2738us) server_negotiation.cc:407] Connection header received
0924 10:44:17.821347 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.821350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.821411 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.821462 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.821935 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50000: BlockingRecv error: recv got EOF from 127.0.0.1:50000 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 10:44:17.823426 24939 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:17.818963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.2.8.60:45923)
0924 10:44:17.819111 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.819117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.822486 (+ 3369us) server_negotiation.cc:407] Connection header received
0924 10:44:17.822734 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.822737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.822787 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.822836 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.823294 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44176: BlockingRecv error: recv got EOF from 127.0.0.1:44176 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 10:44:17.827157 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:17.825702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.2.8.61:38957)
0924 10:44:17.825804 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.825807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.826233 (+ 426us) server_negotiation.cc:407] Connection header received
0924 10:44:17.826452 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.826455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.826527 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.826576 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.827058 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50014: BlockingRecv error: recv got EOF from 127.0.0.1:50014 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 10:44:17.832057 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:17.830673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50030 (local address 127.2.8.61:38957)
0924 10:44:17.830771 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.830775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.831222 (+ 447us) server_negotiation.cc:407] Connection header received
0924 10:44:17.831413 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.831416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.831462 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.831511 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.831963 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50030: BlockingRecv error: recv got EOF from 127.0.0.1:50030 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 10:44:17.837607 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:17.836188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50044 (local address 127.2.8.61:38957)
0924 10:44:17.836284 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.836288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.836741 (+ 453us) server_negotiation.cc:407] Connection header received
0924 10:44:17.836913 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.836916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.836961 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.837035 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.837509 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50044: BlockingRecv error: recv got EOF from 127.0.0.1:50044 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 10:44:17.845409 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:17.843796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50060 (local address 127.2.8.61:38957)
0924 10:44:17.843938 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.843942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.844462 (+ 520us) server_negotiation.cc:407] Connection header received
0924 10:44:17.844654 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.844658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.844706 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.844756 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.845290 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50060: BlockingRecv error: recv got EOF from 127.0.0.1:50060 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 10:44:17.857872 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:17.856020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.2.8.61:38957)
0924 10:44:17.856147 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.856152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.856719 (+ 567us) server_negotiation.cc:407] Connection header received
0924 10:44:17.856950 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.856954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.857035 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.857094 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.857735 (+ 641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50062: BlockingRecv error: recv got EOF from 127.0.0.1:50062 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 10:44:17.878159 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:17.876547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.2.8.61:38957)
0924 10:44:17.876692 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.876697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.877072 (+ 375us) server_negotiation.cc:407] Connection header received
0924 10:44:17.877296 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.877299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.877359 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.877419 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.878028 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50074: BlockingRecv error: recv got EOF from 127.0.0.1:50074 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 10:44:17.913910 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:17.912582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.2.8.61:38957)
0924 10:44:17.912711 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.912716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.912989 (+ 273us) server_negotiation.cc:407] Connection header received
0924 10:44:17.913158 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.913167 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.913217 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.913310 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.913790 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50084: BlockingRecv error: recv got EOF from 127.0.0.1:50084 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 10:44:17.981297 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:17.980058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50090 (local address 127.2.8.61:38957)
0924 10:44:17.980221 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:17.980226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:17.980357 (+ 131us) server_negotiation.cc:407] Connection header received
0924 10:44:17.980543 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:17.980546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:17.980603 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:17.980697 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:17.981190 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50090: BlockingRecv error: recv got EOF from 127.0.0.1:50090 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 10:44:18.112452 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:18.111444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.2.8.61:38957)
0924 10:44:18.111582 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:18.111586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:18.111676 (+ 90us) server_negotiation.cc:407] Connection header received
0924 10:44:18.111803 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:18.111806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:18.111855 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:18.111940 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:18.112342 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50098: BlockingRecv error: recv got EOF from 127.0.0.1:50098 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 10:44:18.371903 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 10:44:18.370635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50108 (local address 127.2.8.61:38957)
0924 10:44:18.370757 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:18.370763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:18.370958 (+ 195us) server_negotiation.cc:407] Connection header received
0924 10:44:18.371134 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:18.371137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:18.371195 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:18.371285 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 10:44:18.371788 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50108: BlockingRecv error: recv got EOF from 127.0.0.1:50108 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 10:44:18.688344 24972 catalog_manager.cc:1259] Loaded cluster ID: 297d0c1b956a4ce0b81e246a2bc1ee2d
I20260924 10:44:18.688426 24972 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01: loading cluster ID for follower catalog manager: success
I20260924 10:44:18.690305 24972 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01: acquiring CA information for follower catalog manager: success
I20260924 10:44:18.691531 24972 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dca4dc1d229140dc91c2146441fabb01: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:44:18.704174 24991 catalog_manager.cc:1259] Loaded cluster ID: 297d0c1b956a4ce0b81e246a2bc1ee2d
I20260924 10:44:18.704257 24991 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011: loading cluster ID for follower catalog manager: success
I20260924 10:44:18.705930 24991 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011: acquiring CA information for follower catalog manager: success
I20260924 10:44:18.707093 24991 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 66800f30d6a641e18ffb243fdf3d5011: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 10:44:18.890038 25066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:18.886161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50120 (local address 127.2.8.61:38957)
0924 10:44:18.886406 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:18.886410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:18.886428 (+ 18us) server_negotiation.cc:407] Connection header received
0924 10:44:18.886529 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:18.886531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:18.886652 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:18.886761 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:18.886996 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:18.887603 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:18.888524 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:18.888740 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:18.888825 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:18.889130 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:18.889569 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:18.889839 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:18.889870 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 10:44:18.889916 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":94,"threads_started":1}
I20260924 10:44:18.896514 25072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:18.892903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.2.8.60:45923)
0924 10:44:18.893168 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:18.893175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:18.893192 (+ 17us) server_negotiation.cc:407] Connection header received
0924 10:44:18.893390 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:18.893393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:18.893499 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:18.893617 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:18.893831 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:18.894335 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:18.895143 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:18.895382 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:18.895490 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:18.895770 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:18.896143 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:18.896271 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:18.896316 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 10:44:18.896365 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":101,"threads_started":1}
I20260924 10:44:18.903204 25072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:18.899643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44194 (local address 127.2.8.60:45923)
0924 10:44:18.899760 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:18.899765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:18.899842 (+ 77us) server_negotiation.cc:407] Connection header received
0924 10:44:18.900012 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:18.900018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:18.900067 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:18.900131 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:18.900479 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:18.901223 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:18.902060 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:18.902381 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:18.902476 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:18.902703 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:18.902772 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:18.902835 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:18.902994 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 10:44:18.903052 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 10:44:18.903342 25066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:18.899244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.2.8.61:38957)
0924 10:44:18.899361 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:18.899364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:18.899684 (+ 320us) server_negotiation.cc:407] Connection header received
0924 10:44:18.899841 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:18.899845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:18.899896 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:18.899963 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:18.900279 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:18.901121 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:18.902210 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:18.902551 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:18.902623 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:18.902878 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:18.902964 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:18.903030 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:18.903171 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 10:44:18.903220 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 10:44:18.904408 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 10:44:18.899202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.2.8.62:32993)
0924 10:44:18.899462 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 10:44:18.899467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 10:44:18.899740 (+ 273us) server_negotiation.cc:407] Connection header received
0924 10:44:18.899847 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 10:44:18.899849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 10:44:18.900008 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 10:44:18.900141 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 10:44:18.900393 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:18.900937 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:18.902712 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 10:44:18.903076 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 10:44:18.903168 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 10:44:18.903664 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 10:44:18.903960 (+ 296us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 10:44:18.904044 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 10:44:18.904183 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 10:44:18.904245 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":54,"threads_started":1}
I20260924 10:44:18.907147 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 24715
I20260924 10:44:18.913081 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 24790
I20260924 10:44:18.918449 2080 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2SD_Oz/build/debug/bin/kudu with pid 24865
2026-09-24T10:44:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3066 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6143 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (284091 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST